2024-08-19T10:59:54.659 INFO:root:teuthology version: 1.2.0 2024-08-19T10:59:54.671 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-19T10:59:54.760 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859430 branch: main description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} email: ceph-qa@ceph.com first_in_suite: false job_id: '7859430' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b workunit: branch: main sha1: 964adda23da9f7b057cb34fdc0b82aeb7ed65324 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1858 sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b sleep_before_teardown: 0 subset: 29813/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 964adda23da9f7b057cb34fdc0b82aeb7ed65324 targets: smithi130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJtFBKmgyM6YnfzVxKPaLPPMpDN5QQTeOq+gWuOBtTD4111jBWuB5mi0yP6FfMZ30DYb/eiuPAw4HKNA1CoO2L0= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-18_20:00:17 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-19T10:59:54.760 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa; will attempt to use it 2024-08-19T10:59:54.760 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks 2024-08-19T10:59:54.761 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-19T10:59:54.762 INFO:teuthology.task.internal:Checking packages... 2024-08-19T10:59:54.784 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '797669399b32d2915e64eb6d4b1257ae3c50f80b' 2024-08-19T10:59:54.784 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T10:59:54.784 INFO:teuthology.packaging:ref: None 2024-08-19T10:59:54.784 INFO:teuthology.packaging:tag: None 2024-08-19T10:59:54.784 INFO:teuthology.packaging:branch: main 2024-08-19T10:59:54.784 INFO:teuthology.packaging:sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-19T10:59:54.784 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-08-19T10:59:55.033 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5847-g79766939-1jammy 2024-08-19T10:59:55.034 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-19T10:59:55.041 INFO:teuthology.task.internal:no buildpackages task found 2024-08-19T10:59:55.041 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-19T10:59:55.188 INFO:teuthology.task.internal:Saving configuration 2024-08-19T10:59:55.197 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-19T10:59:55.248 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-19T10:59:55.282 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859430', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-19 10:54:28.825734', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILNBpJKxRO/7AQcQXNeLQUhDjo1SienQdeXjxwavH44U'} 2024-08-19T10:59:55.282 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-19T10:59:55.288 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-08-19T10:59:55.288 INFO:teuthology.run_tasks:Running task console_log... 2024-08-19T10:59:55.335 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f89fde26e60>, signals=[15]) 2024-08-19T10:59:55.335 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-19T10:59:55.340 INFO:teuthology.task.internal:Opening connections... 2024-08-19T10:59:55.340 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2024-08-19T10:59:55.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T10:59:55.433 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-19T10:59:55.441 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2024-08-19T10:59:55.530 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2024-08-19T10:59:55.531 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2024-08-19T10:59:55.576 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-08-19T10:59:55.576 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="Ubuntu" 2024-08-19T10:59:55.576 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="22.04" 2024-08-19T10:59:55.576 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-08-19T10:59:55.576 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_CODENAME=jammy 2024-08-19T10:59:55.576 INFO:teuthology.orchestra.run.smithi130.stdout:ID=ubuntu 2024-08-19T10:59:55.577 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE=debian 2024-08-19T10:59:55.577 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://www.ubuntu.com/" 2024-08-19T10:59:55.577 INFO:teuthology.orchestra.run.smithi130.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-08-19T10:59:55.577 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-08-19T10:59:55.577 INFO:teuthology.orchestra.run.smithi130.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-08-19T10:59:55.577 INFO:teuthology.orchestra.run.smithi130.stdout:UBUNTU_CODENAME=jammy 2024-08-19T10:59:55.577 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2024-08-19T10:59:55.604 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-19T10:59:55.612 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-19T10:59:55.745 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-19T10:59:55.745 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2024-08-19T10:59:55.750 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-19T10:59:55.820 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-19T10:59:55.821 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2024-08-19T10:59:55.942 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-19T10:59:55.983 INFO:teuthology.run_tasks:Running task kernel... 2024-08-19T10:59:56.032 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-19T10:59:56.032 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-19T10:59:56.032 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-19T10:59:56.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T10:59:56.037 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2024-08-19T10:59:56.085 INFO:teuthology.orchestra.run.smithi130.stdout:5.15.0-56-generic 2024-08-19T10:59:56.085 INFO:teuthology.task.kernel:Running kernel on smithi130: 5.15.0-56-generic 2024-08-19T10:59:56.086 DEBUG:teuthology.orchestra.run.smithi130:> sudo apt-get clean 2024-08-19T10:59:56.238 DEBUG:teuthology.orchestra.run.smithi130:> sudo apt-get update 2024-08-19T10:59:56.381 INFO:teuthology.orchestra.run.smithi130.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-08-19T10:59:56.490 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-19T10:59:56.572 INFO:teuthology.orchestra.run.smithi130.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-08-19T10:59:56.598 INFO:teuthology.orchestra.run.smithi130.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [524 kB] 2024-08-19T10:59:56.667 INFO:teuthology.orchestra.run.smithi130.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,729 kB] 2024-08-19T10:59:56.704 INFO:teuthology.orchestra.run.smithi130.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [287 kB] 2024-08-19T10:59:56.707 INFO:teuthology.orchestra.run.smithi130.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.2 kB] 2024-08-19T10:59:56.707 INFO:teuthology.orchestra.run.smithi130.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.3 kB] 2024-08-19T10:59:56.717 INFO:teuthology.orchestra.run.smithi130.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,247 kB] 2024-08-19T10:59:56.734 INFO:teuthology.orchestra.run.smithi130.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [388 kB] 2024-08-19T10:59:56.737 INFO:teuthology.orchestra.run.smithi130.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [572 B] 2024-08-19T10:59:56.737 INFO:teuthology.orchestra.run.smithi130.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [890 kB] 2024-08-19T10:59:56.743 INFO:teuthology.orchestra.run.smithi130.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [621 kB] 2024-08-19T10:59:56.747 INFO:teuthology.orchestra.run.smithi130.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [175 kB] 2024-08-19T10:59:56.749 INFO:teuthology.orchestra.run.smithi130.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.0 kB] 2024-08-19T10:59:56.766 INFO:teuthology.orchestra.run.smithi130.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-08-19T10:59:56.796 INFO:teuthology.orchestra.run.smithi130.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-08-19T10:59:56.798 INFO:teuthology.orchestra.run.smithi130.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-08-19T10:59:56.798 INFO:teuthology.orchestra.run.smithi130.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-08-19T10:59:56.928 INFO:teuthology.orchestra.run.smithi130.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-08-19T10:59:57.067 INFO:teuthology.orchestra.run.smithi130.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,948 kB] 2024-08-19T10:59:57.379 INFO:teuthology.orchestra.run.smithi130.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [684 kB] 2024-08-19T10:59:57.416 INFO:teuthology.orchestra.run.smithi130.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [345 kB] 2024-08-19T10:59:57.431 INFO:teuthology.orchestra.run.smithi130.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-08-19T10:59:57.432 INFO:teuthology.orchestra.run.smithi130.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.9 kB] 2024-08-19T10:59:57.433 INFO:teuthology.orchestra.run.smithi130.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,322 kB] 2024-08-19T10:59:57.498 INFO:teuthology.orchestra.run.smithi130.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [401 kB] 2024-08-19T10:59:57.512 INFO:teuthology.orchestra.run.smithi130.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [604 B] 2024-08-19T10:59:57.512 INFO:teuthology.orchestra.run.smithi130.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [725 kB] 2024-08-19T10:59:57.535 INFO:teuthology.orchestra.run.smithi130.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,111 kB] 2024-08-19T10:59:57.565 INFO:teuthology.orchestra.run.smithi130.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [259 kB] 2024-08-19T10:59:57.571 INFO:teuthology.orchestra.run.smithi130.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [25.9 kB] 2024-08-19T10:59:57.573 INFO:teuthology.orchestra.run.smithi130.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-08-19T10:59:57.574 INFO:teuthology.orchestra.run.smithi130.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-08-19T10:59:57.575 INFO:teuthology.orchestra.run.smithi130.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-08-19T10:59:57.623 INFO:teuthology.orchestra.run.smithi130.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-08-19T10:59:57.623 INFO:teuthology.orchestra.run.smithi130.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-08-19T10:59:57.625 INFO:teuthology.orchestra.run.smithi130.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-08-19T10:59:57.626 INFO:teuthology.orchestra.run.smithi130.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-08-19T10:59:57.704 INFO:teuthology.orchestra.run.smithi130.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-08-19T10:59:57.705 INFO:teuthology.orchestra.run.smithi130.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-08-19T10:59:57.705 INFO:teuthology.orchestra.run.smithi130.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-08-19T10:59:57.706 INFO:teuthology.orchestra.run.smithi130.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-08-19T10:59:57.706 INFO:teuthology.orchestra.run.smithi130.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-08-19T11:00:02.974 INFO:teuthology.orchestra.run.smithi130.stdout:Fetched 15.5 MB in 2s (7,465 kB/s) 2024-08-19T11:00:04.379 INFO:teuthology.orchestra.run.smithi130.stdout:Reading package lists... 2024-08-19T11:00:04.397 DEBUG:teuthology.orchestra.run.smithi130:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-08-19T11:00:04.461 INFO:teuthology.orchestra.run.smithi130.stdout:Reading package lists... 2024-08-19T11:00:04.678 INFO:teuthology.orchestra.run.smithi130.stdout:Building dependency tree... 2024-08-19T11:00:04.679 INFO:teuthology.orchestra.run.smithi130.stdout:Reading state information... 2024-08-19T11:00:04.872 INFO:teuthology.orchestra.run.smithi130.stdout:The following packages were automatically installed and are no longer required: 2024-08-19T11:00:04.872 INFO:teuthology.orchestra.run.smithi130.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-19T11:00:04.872 INFO:teuthology.orchestra.run.smithi130.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-19T11:00:04.872 INFO:teuthology.orchestra.run.smithi130.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-19T11:00:04.873 INFO:teuthology.orchestra.run.smithi130.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-08-19T11:00:04.873 INFO:teuthology.orchestra.run.smithi130.stdout: python2.7-minimal 2024-08-19T11:00:04.873 INFO:teuthology.orchestra.run.smithi130.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-19T11:00:04.875 INFO:teuthology.orchestra.run.smithi130.stdout:The following additional packages will be installed: 2024-08-19T11:00:04.875 INFO:teuthology.orchestra.run.smithi130.stdout: linux-generic linux-headers-5.15.0-118 linux-headers-5.15.0-118-generic 2024-08-19T11:00:04.875 INFO:teuthology.orchestra.run.smithi130.stdout: linux-headers-generic linux-image-5.15.0-118-generic 2024-08-19T11:00:04.875 INFO:teuthology.orchestra.run.smithi130.stdout: linux-modules-5.15.0-118-generic linux-modules-extra-5.15.0-118-generic 2024-08-19T11:00:04.877 INFO:teuthology.orchestra.run.smithi130.stdout:Suggested packages: 2024-08-19T11:00:04.877 INFO:teuthology.orchestra.run.smithi130.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-08-19T11:00:04.877 INFO:teuthology.orchestra.run.smithi130.stdout:Recommended packages: 2024-08-19T11:00:04.877 INFO:teuthology.orchestra.run.smithi130.stdout: thermald 2024-08-19T11:00:04.921 INFO:teuthology.orchestra.run.smithi130.stdout:The following NEW packages will be installed: 2024-08-19T11:00:04.921 INFO:teuthology.orchestra.run.smithi130.stdout: linux-headers-5.15.0-118 linux-headers-5.15.0-118-generic 2024-08-19T11:00:04.921 INFO:teuthology.orchestra.run.smithi130.stdout: linux-image-5.15.0-118-generic linux-modules-5.15.0-118-generic 2024-08-19T11:00:04.921 INFO:teuthology.orchestra.run.smithi130.stdout: linux-modules-extra-5.15.0-118-generic 2024-08-19T11:00:04.923 INFO:teuthology.orchestra.run.smithi130.stdout:The following packages will be upgraded: 2024-08-19T11:00:04.924 INFO:teuthology.orchestra.run.smithi130.stdout: linux-generic linux-headers-generic linux-image-generic 2024-08-19T11:00:09.463 INFO:teuthology.orchestra.run.smithi130.stdout:3 upgraded, 5 newly installed, 0 to remove and 333 not upgraded. 2024-08-19T11:00:09.463 INFO:teuthology.orchestra.run.smithi130.stdout:Need to get 113 MB of archives. 2024-08-19T11:00:09.463 INFO:teuthology.orchestra.run.smithi130.stdout:After this operation, 583 MB of additional disk space will be used. 2024-08-19T11:00:09.463 INFO:teuthology.orchestra.run.smithi130.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-118-generic amd64 5.15.0-118.128 [22.6 MB] 2024-08-19T11:00:09.856 INFO:teuthology.orchestra.run.smithi130.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-118-generic amd64 5.15.0-118.128 [11.5 MB] 2024-08-19T11:00:09.945 INFO:teuthology.orchestra.run.smithi130.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-118-generic amd64 5.15.0-118.128 [64.0 MB] 2024-08-19T11:00:10.729 INFO:teuthology.orchestra.run.smithi130.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.118.118 [1,694 B] 2024-08-19T11:00:10.729 INFO:teuthology.orchestra.run.smithi130.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.118.118 [2,466 B] 2024-08-19T11:00:10.730 INFO:teuthology.orchestra.run.smithi130.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-118 all 5.15.0-118.128 [12.4 MB] 2024-08-19T11:00:10.848 INFO:teuthology.orchestra.run.smithi130.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-118-generic amd64 5.15.0-118.128 [2,876 kB] 2024-08-19T11:00:10.869 INFO:teuthology.orchestra.run.smithi130.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.118.118 [2,322 B] 2024-08-19T11:00:11.348 INFO:teuthology.orchestra.run.smithi130.stdout:Fetched 113 MB in 6s (18.9 MB/s) 2024-08-19T11:00:11.542 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package linux-modules-5.15.0-118-generic. 2024-08-19T11:00:13.572 INFO:teuthology.orchestra.run.smithi130.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-08-19T11:00:13.576 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../0-linux-modules-5.15.0-118-generic_5.15.0-118.128_amd64.deb ... 2024-08-19T11:00:13.781 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-modules-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-19T11:00:15.908 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package linux-image-5.15.0-118-generic. 2024-08-19T11:00:15.929 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../1-linux-image-5.15.0-118-generic_5.15.0-118.128_amd64.deb ... 2024-08-19T11:00:16.021 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-image-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-19T11:00:16.428 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-118-generic. 2024-08-19T11:00:16.449 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-118-generic_5.15.0-118.128_amd64.deb ... 2024-08-19T11:00:16.495 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-modules-extra-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-19T11:00:23.530 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../3-linux-generic_5.15.0.118.118_amd64.deb ... 2024-08-19T11:00:23.657 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-generic (5.15.0.118.118) over (5.15.0.56.54) ... 2024-08-19T11:00:23.990 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.118.118_amd64.deb ... 2024-08-19T11:00:24.094 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-image-generic (5.15.0.118.118) over (5.15.0.56.54) ... 2024-08-19T11:00:24.416 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package linux-headers-5.15.0-118. 2024-08-19T11:00:24.438 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../5-linux-headers-5.15.0-118_5.15.0-118.128_all.deb ... 2024-08-19T11:00:24.482 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-headers-5.15.0-118 (5.15.0-118.128) ... 2024-08-19T11:00:30.399 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package linux-headers-5.15.0-118-generic. 2024-08-19T11:00:30.436 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../6-linux-headers-5.15.0-118-generic_5.15.0-118.128_amd64.deb ... 2024-08-19T11:00:30.473 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-headers-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-19T11:00:34.159 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.118.118_amd64.deb ... 2024-08-19T11:00:34.276 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-headers-generic (5.15.0.118.118) over (5.15.0.56.54) ... 2024-08-19T11:00:34.635 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-headers-5.15.0-118 (5.15.0-118.128) ... 2024-08-19T11:00:34.740 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-headers-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-19T11:00:34.866 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-headers-generic (5.15.0.118.118) ... 2024-08-19T11:00:35.003 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-image-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-19T11:00:36.339 INFO:teuthology.orchestra.run.smithi130.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-08-19T11:00:36.339 INFO:teuthology.orchestra.run.smithi130.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-08-19T11:00:36.339 INFO:teuthology.orchestra.run.smithi130.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-118-generic 2024-08-19T11:00:36.339 INFO:teuthology.orchestra.run.smithi130.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-118-generic 2024-08-19T11:00:36.505 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-modules-extra-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-19T11:00:37.882 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-modules-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-19T11:00:39.358 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-image-generic (5.15.0.118.118) ... 2024-08-19T11:00:39.483 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-generic (5.15.0.118.118) ... 2024-08-19T11:00:39.610 INFO:teuthology.orchestra.run.smithi130.stdout:Processing triggers for linux-image-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-19T11:00:39.679 INFO:teuthology.orchestra.run.smithi130.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-08-19T11:00:39.680 INFO:teuthology.orchestra.run.smithi130.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-118-generic 2024-08-19T11:00:53.660 INFO:teuthology.orchestra.run.smithi130.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-08-19T11:00:53.661 INFO:teuthology.orchestra.run.smithi130.stdout:Sourcing file `/etc/default/grub' 2024-08-19T11:00:53.681 INFO:teuthology.orchestra.run.smithi130.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-08-19T11:00:53.701 INFO:teuthology.orchestra.run.smithi130.stdout:Generating grub configuration file ... 2024-08-19T11:00:53.946 INFO:teuthology.orchestra.run.smithi130.stdout:Found linux image: /boot/vmlinuz-5.15.0-118-generic 2024-08-19T11:00:53.960 INFO:teuthology.orchestra.run.smithi130.stdout:Found initrd image: /boot/initrd.img-5.15.0-118-generic 2024-08-19T11:00:54.242 INFO:teuthology.orchestra.run.smithi130.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-08-19T11:00:54.247 INFO:teuthology.orchestra.run.smithi130.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-08-19T11:00:54.319 INFO:teuthology.orchestra.run.smithi130.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-08-19T11:00:54.324 INFO:teuthology.orchestra.run.smithi130.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-08-19T11:00:54.516 INFO:teuthology.orchestra.run.smithi130.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-08-19T11:00:54.516 INFO:teuthology.orchestra.run.smithi130.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-08-19T11:00:54.516 INFO:teuthology.orchestra.run.smithi130.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-08-19T11:00:54.547 INFO:teuthology.orchestra.run.smithi130.stdout:done 2024-08-19T11:00:55.006 DEBUG:teuthology.orchestra.run.smithi130:> dpkg -s linux-image-generic 2024-08-19T11:00:55.028 INFO:teuthology.orchestra.run.smithi130.stdout:Package: linux-image-generic 2024-08-19T11:00:55.028 INFO:teuthology.orchestra.run.smithi130.stdout:Status: install ok installed 2024-08-19T11:00:55.028 INFO:teuthology.orchestra.run.smithi130.stdout:Priority: optional 2024-08-19T11:00:55.028 INFO:teuthology.orchestra.run.smithi130.stdout:Section: kernel 2024-08-19T11:00:55.028 INFO:teuthology.orchestra.run.smithi130.stdout:Installed-Size: 21 2024-08-19T11:00:55.028 INFO:teuthology.orchestra.run.smithi130.stdout:Maintainer: Ubuntu Kernel Team 2024-08-19T11:00:55.029 INFO:teuthology.orchestra.run.smithi130.stdout:Architecture: amd64 2024-08-19T11:00:55.029 INFO:teuthology.orchestra.run.smithi130.stdout:Source: linux-meta 2024-08-19T11:00:55.029 INFO:teuthology.orchestra.run.smithi130.stdout:Version: 5.15.0.118.118 2024-08-19T11:00:55.029 INFO:teuthology.orchestra.run.smithi130.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-118), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-08-19T11:00:55.029 INFO:teuthology.orchestra.run.smithi130.stdout:Depends: linux-image-5.15.0-118-generic, linux-modules-extra-5.15.0-118-generic, linux-firmware, intel-microcode, amd64-microcode 2024-08-19T11:00:55.029 INFO:teuthology.orchestra.run.smithi130.stdout:Recommends: thermald 2024-08-19T11:00:55.029 INFO:teuthology.orchestra.run.smithi130.stdout:Description: Generic Linux kernel image 2024-08-19T11:00:55.029 INFO:teuthology.orchestra.run.smithi130.stdout: This package will always depend on the latest generic kernel image 2024-08-19T11:00:55.029 INFO:teuthology.orchestra.run.smithi130.stdout: available. 2024-08-19T11:00:55.029 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-118-generic 2024-08-19T11:00:55.030 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-19T11:00:55.030 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-19T11:00:55.030 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-08-19T11:00:55.030 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi130.front.sepia.ceph.com, path=None, version=distro) 2024-08-19T11:00:55.030 DEBUG:teuthology.orchestra.run.smithi130:> sudo apt-get clean 2024-08-19T11:00:55.221 DEBUG:teuthology.orchestra.run.smithi130:> sudo apt-get update 2024-08-19T11:00:55.347 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-19T11:00:55.347 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-08-19T11:00:55.356 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-08-19T11:00:55.373 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-08-19T11:00:56.777 INFO:teuthology.orchestra.run.smithi130.stdout:Reading package lists... 2024-08-19T11:00:56.795 DEBUG:teuthology.orchestra.run.smithi130:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-08-19T11:00:56.859 INFO:teuthology.orchestra.run.smithi130.stdout:Reading package lists... 2024-08-19T11:00:57.066 INFO:teuthology.orchestra.run.smithi130.stdout:Building dependency tree... 2024-08-19T11:00:57.067 INFO:teuthology.orchestra.run.smithi130.stdout:Reading state information... 2024-08-19T11:00:57.241 INFO:teuthology.orchestra.run.smithi130.stdout:linux-image-generic is already the newest version (5.15.0.118.118). 2024-08-19T11:00:57.241 INFO:teuthology.orchestra.run.smithi130.stdout:The following packages were automatically installed and are no longer required: 2024-08-19T11:00:57.241 INFO:teuthology.orchestra.run.smithi130.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-19T11:00:57.241 INFO:teuthology.orchestra.run.smithi130.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-19T11:00:57.241 INFO:teuthology.orchestra.run.smithi130.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-19T11:00:57.242 INFO:teuthology.orchestra.run.smithi130.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-08-19T11:00:57.242 INFO:teuthology.orchestra.run.smithi130.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-08-19T11:00:57.242 INFO:teuthology.orchestra.run.smithi130.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-19T11:00:57.281 INFO:teuthology.orchestra.run.smithi130.stdout:0 upgraded, 0 newly installed, 0 to remove and 333 not upgraded. 2024-08-19T11:00:57.283 DEBUG:teuthology.orchestra.run.smithi130:> dpkg -s linux-image-generic 2024-08-19T11:00:57.295 INFO:teuthology.orchestra.run.smithi130.stdout:Package: linux-image-generic 2024-08-19T11:00:57.295 INFO:teuthology.orchestra.run.smithi130.stdout:Status: install ok installed 2024-08-19T11:00:57.295 INFO:teuthology.orchestra.run.smithi130.stdout:Priority: optional 2024-08-19T11:00:57.295 INFO:teuthology.orchestra.run.smithi130.stdout:Section: kernel 2024-08-19T11:00:57.295 INFO:teuthology.orchestra.run.smithi130.stdout:Installed-Size: 21 2024-08-19T11:00:57.295 INFO:teuthology.orchestra.run.smithi130.stdout:Maintainer: Ubuntu Kernel Team 2024-08-19T11:00:57.295 INFO:teuthology.orchestra.run.smithi130.stdout:Architecture: amd64 2024-08-19T11:00:57.295 INFO:teuthology.orchestra.run.smithi130.stdout:Source: linux-meta 2024-08-19T11:00:57.295 INFO:teuthology.orchestra.run.smithi130.stdout:Version: 5.15.0.118.118 2024-08-19T11:00:57.296 INFO:teuthology.orchestra.run.smithi130.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-118), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-08-19T11:00:57.296 INFO:teuthology.orchestra.run.smithi130.stdout:Depends: linux-image-5.15.0-118-generic, linux-modules-extra-5.15.0-118-generic, linux-firmware, intel-microcode, amd64-microcode 2024-08-19T11:00:57.296 INFO:teuthology.orchestra.run.smithi130.stdout:Recommends: thermald 2024-08-19T11:00:57.296 INFO:teuthology.orchestra.run.smithi130.stdout:Description: Generic Linux kernel image 2024-08-19T11:00:57.296 INFO:teuthology.orchestra.run.smithi130.stdout: This package will always depend on the latest generic kernel image 2024-08-19T11:00:57.296 INFO:teuthology.orchestra.run.smithi130.stdout: available. 2024-08-19T11:00:57.296 DEBUG:teuthology.orchestra.run.smithi130:> mktemp 2024-08-19T11:00:57.340 INFO:teuthology.orchestra.run.smithi130.stdout:/tmp/tmp.KGWwYThmGA 2024-08-19T11:00:57.341 DEBUG:teuthology.orchestra.run.smithi130:> sudo cp /boot/grub/grub.cfg /tmp/tmp.KGWwYThmGA 2024-08-19T11:00:57.396 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0666 /tmp/tmp.KGWwYThmGA 2024-08-19T11:00:57.513 DEBUG:teuthology.orchestra.remote:smithi130:/tmp/tmp.KGWwYThmGA is 10KB 2024-08-19T11:00:57.567 DEBUG:teuthology.orchestra.run.smithi130:> rm -fr /tmp/tmp.KGWwYThmGA 2024-08-19T11:00:57.572 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-08-19T11:00:57.624 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-08-19T11:00:57.624 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-08-19T11:00:57.624 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-08-19T11:00:57.688 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-118-generic 2024-08-19T11:00:57.688 DEBUG:teuthology.orchestra.run.smithi130:> sudo update-grub 2024-08-19T11:00:58.842 INFO:teuthology.orchestra.run.smithi130.stderr:Sourcing file `/etc/default/grub' 2024-08-19T11:00:58.844 INFO:teuthology.orchestra.run.smithi130.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-08-19T11:00:58.851 INFO:teuthology.orchestra.run.smithi130.stderr:Generating grub configuration file ... 2024-08-19T11:00:59.023 INFO:teuthology.orchestra.run.smithi130.stderr:Found linux image: /boot/vmlinuz-5.15.0-118-generic 2024-08-19T11:00:59.028 INFO:teuthology.orchestra.run.smithi130.stderr:Found initrd image: /boot/initrd.img-5.15.0-118-generic 2024-08-19T11:00:59.246 INFO:teuthology.orchestra.run.smithi130.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-08-19T11:00:59.251 INFO:teuthology.orchestra.run.smithi130.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-08-19T11:00:59.288 INFO:teuthology.orchestra.run.smithi130.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-08-19T11:00:59.293 INFO:teuthology.orchestra.run.smithi130.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-08-19T11:00:59.436 INFO:teuthology.orchestra.run.smithi130.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-08-19T11:00:59.436 INFO:teuthology.orchestra.run.smithi130.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-08-19T11:00:59.436 INFO:teuthology.orchestra.run.smithi130.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-08-19T11:00:59.445 INFO:teuthology.orchestra.run.smithi130.stderr:done 2024-08-19T11:00:59.458 DEBUG:teuthology.orchestra.run.smithi130:> sudo shutdown -r now 2024-08-19T11:01:29.486 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-08-19T11:01:29.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-08-19T11:01:29.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T11:01:47.875 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.130 2024-08-19T11:01:56.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-08-19T11:01:56.884 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T11:01:59.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.130 2024-08-19T11:02:11.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-08-19T11:02:11.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T11:02:30.367 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.130 2024-08-19T11:02:45.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-08-19T11:02:45.384 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T11:02:45.943 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-08-19T11:02:47.359 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-08-19T11:02:47.360 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-118-generic"... 2024-08-19T11:02:47.360 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2024-08-19T11:02:47.405 INFO:teuthology.orchestra.run.smithi130.stdout:5.15.0-118-generic 2024-08-19T11:02:47.405 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-118-generic vs 5.15.0-118-generic 2024-08-19T11:02:47.405 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-19T11:02:47.405 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-08-19T11:02:48.406 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-19T11:02:48.406 DEBUG:teuthology.orchestra.run.smithi130:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-19T11:02:48.544 INFO:teuthology.orchestra.run.smithi130.stdout:ttyS1 2024-08-19T11:02:48.567 DEBUG:teuthology.parallel:result is None 2024-08-19T11:02:48.567 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-19T11:02:48.574 INFO:teuthology.task.internal:Creating test directory... 2024-08-19T11:02:48.575 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-19T11:02:48.579 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-19T11:02:48.631 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-19T11:02:48.717 INFO:teuthology.task.internal:Creating archive directory... 2024-08-19T11:02:48.717 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-19T11:02:48.832 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-19T11:02:48.838 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-19T11:02:48.839 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-19T11:02:48.925 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:02:48.933 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:02:48.935 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-19T11:02:48.942 INFO:teuthology.task.internal:Configuring sudo... 2024-08-19T11:02:48.943 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-19T11:02:48.957 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-19T11:02:49.041 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-19T11:02:49.042 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-19T11:02:49.048 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T11:02:49.093 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T11:02:49.141 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-08-19T11:02:49.141 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-19T11:02:49.204 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2024-08-19T11:02:49.288 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-19T11:02:49.296 INFO:teuthology.task.internal:Starting timer... 2024-08-19T11:02:49.296 INFO:teuthology.run_tasks:Running task pcp... 2024-08-19T11:02:49.309 INFO:teuthology.run_tasks:Running task selinux... 2024-08-19T11:02:49.317 DEBUG:teuthology.task.selinux:Excluding smithi130: OS 'ubuntu' does not support SELinux 2024-08-19T11:02:49.318 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-19T11:02:49.318 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-08-19T11:02:49.318 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-19T11:02:49.318 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-19T11:02:49.327 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-19T11:02:49.339 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-19T11:02:49.340 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi130.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-19T11:10:09.361 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi130.front.sepia.ceph.com')] 2024-08-19T11:10:09.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-08-19T11:10:09.364 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T11:10:09.460 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-08-19T11:10:09.539 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-08-19T11:10:09.539 INFO:teuthology.run_tasks:Running task clock... 2024-08-19T11:10:09.553 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-19T11:10:09.553 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-19T11:10:09.553 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T11:10:09.613 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-08-19T11:10:09.613 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: Command line: ntpd -gq 2024-08-19T11:10:09.613 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: ---------------------------------------------------- 2024-08-19T11:10:09.613 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: ntp-4 is maintained by Network Time Foundation, 2024-08-19T11:10:09.613 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-08-19T11:10:09.614 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: corporation. Support and training for ntp-4 are 2024-08-19T11:10:09.614 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: available at https://www.nwtime.org/support 2024-08-19T11:10:09.614 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: ---------------------------------------------------- 2024-08-19T11:10:09.614 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: proto: precision = 0.064 usec (-24) 2024-08-19T11:10:09.615 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: basedate set to 2022-02-04 2024-08-19T11:10:09.615 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: gps base set to 2022-02-06 (week 2196) 2024-08-19T11:10:09.615 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-08-19T11:10:09.615 INFO:teuthology.orchestra.run.smithi130.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-08-19T11:10:09.615 INFO:teuthology.orchestra.run.smithi130.stderr:restrict ::: KOD does nothing without LIMITED. 2024-08-19T11:10:09.616 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: restrict ::: KOD does nothing without LIMITED. 2024-08-19T11:10:09.616 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: Listen and drop on 0 v6wildcard [::]:123 2024-08-19T11:10:09.616 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-08-19T11:10:09.616 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: Listen normally on 2 lo 127.0.0.1:123 2024-08-19T11:10:09.616 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: Listen normally on 3 enp3s0f1 172.21.15.130:123 2024-08-19T11:10:09.616 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: Listen normally on 4 lo [::1]:123 2024-08-19T11:10:09.617 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f37%5]:123 2024-08-19T11:10:09.617 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:09 ntpd[18360]: Listening on routing socket on fd #22 for interface updates 2024-08-19T11:10:12.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:12 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:12.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:12 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:13.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:13 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:13.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:13 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:13.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:13 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:13.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:13 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:13.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:13 ntpd[18360]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-19T11:10:13.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:13 ntpd[18360]: can't open /var/log/ntpstats/peerstats.20240819: Permission denied 2024-08-19T11:10:14.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:14 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:14.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:14 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:14.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:14 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:14.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:14 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:14.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:14 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:14.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:14 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:14.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:14 ntpd[18360]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-19T11:10:14.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:14 ntpd[18360]: can't open /var/log/ntpstats/peerstats.20240819: Permission denied 2024-08-19T11:10:14.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:14 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:14.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:14 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:14.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:14 ntpd[18360]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-19T11:10:14.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:14 ntpd[18360]: can't open /var/log/ntpstats/peerstats.20240819: Permission denied 2024-08-19T11:10:15.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:15 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:15.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:15 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:15.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:15 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:15.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:15 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:15.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:15 ntpd[18360]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-19T11:10:15.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:15 ntpd[18360]: can't open /var/log/ntpstats/peerstats.20240819: Permission denied 2024-08-19T11:10:16.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:16 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:16.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:16 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:16.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:16 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:16.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:16 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:16.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:16 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:16.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:16 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:16.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:16 ntpd[18360]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-19T11:10:16.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:16 ntpd[18360]: can't open /var/log/ntpstats/peerstats.20240819: Permission denied 2024-08-19T11:10:16.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:16 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:16.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:16 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:16.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:16 ntpd[18360]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-19T11:10:16.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:16 ntpd[18360]: can't open /var/log/ntpstats/peerstats.20240819: Permission denied 2024-08-19T11:10:17.613 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:17 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:17.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:17 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:17.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:17 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:17.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:17 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:17.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:17 ntpd[18360]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-19T11:10:17.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:17 ntpd[18360]: can't open /var/log/ntpstats/peerstats.20240819: Permission denied 2024-08-19T11:10:18.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:18 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:18.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:18 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:18.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:18 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:18.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:18 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:18.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:18 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:18.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:18 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:18.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:18 ntpd[18360]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-19T11:10:18.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:18 ntpd[18360]: can't open /var/log/ntpstats/peerstats.20240819: Permission denied 2024-08-19T11:10:18.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:18 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:18.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:18 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:18.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:18 ntpd[18360]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-19T11:10:18.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:18 ntpd[18360]: can't open /var/log/ntpstats/peerstats.20240819: Permission denied 2024-08-19T11:10:19.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:19 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:19.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:19 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:19.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:19 ntpd[18360]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-19T11:10:19.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:19 ntpd[18360]: can't open /var/log/ntpstats/rawstats.20240819: Permission denied 2024-08-19T11:10:19.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:19 ntpd[18360]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-19T11:10:19.614 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:19 ntpd[18360]: can't open /var/log/ntpstats/peerstats.20240819: Permission denied 2024-08-19T11:10:19.614 INFO:teuthology.orchestra.run.smithi130.stdout:19 Aug 11:10:19 ntpd[18360]: ntpd: time slew -0.001133 s 2024-08-19T11:10:19.615 INFO:teuthology.orchestra.run.smithi130.stdout:ntpd: time slew -0.001133s 2024-08-19T11:10:19.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:19 ntpd[18360]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-08-19T11:10:19.615 INFO:teuthology.orchestra.run.smithi130.stderr:19 Aug 11:10:19 ntpd[18360]: can't open /var/log/ntpstats/loopstats.20240819: Permission denied 2024-08-19T11:10:19.666 INFO:teuthology.orchestra.run.smithi130.stdout: remote refid st t when poll reach delay offset jitter 2024-08-19T11:10:19.666 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================== 2024-08-19T11:10:19.666 INFO:teuthology.orchestra.run.smithi130.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-19T11:10:19.666 INFO:teuthology.orchestra.run.smithi130.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-19T11:10:19.666 INFO:teuthology.orchestra.run.smithi130.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-19T11:10:19.666 INFO:teuthology.orchestra.run.smithi130.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-19T11:10:19.667 INFO:teuthology.run_tasks:Running task install... 2024-08-19T11:10:19.679 DEBUG:teuthology.task.install:project ceph 2024-08-19T11:10:19.679 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b'}} 2024-08-19T11:10:19.679 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b'} 2024-08-19T11:10:19.679 INFO:teuthology.task.install:Using flavor: default 2024-08-19T11:10:19.685 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-19T11:10:19.685 INFO:teuthology.task.install:extra packages: [] 2024-08-19T11:10:19.686 DEBUG:teuthology.orchestra.run.smithi130:> sudo apt-key list | grep Ceph 2024-08-19T11:10:19.771 INFO:teuthology.orchestra.run.smithi130.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-08-19T11:10:19.800 INFO:teuthology.orchestra.run.smithi130.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-08-19T11:10:19.800 INFO:teuthology.orchestra.run.smithi130.stdout:uid [ unknown] Ceph.com (release key) 2024-08-19T11:10:19.800 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-08-19T11:10:19.801 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-19T11:10:19.951 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default/ 2024-08-19T11:10:19.952 INFO:teuthology.task.install.deb:Package version is 19.0.0-5847-g79766939-1jammy 2024-08-19T11:10:20.075 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-08-19T11:10:20.075 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-08-19T11:10:20.090 DEBUG:teuthology.orchestra.run.smithi130:> sudo apt-get update 2024-08-19T11:10:20.262 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-19T11:10:20.273 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-08-19T11:10:20.274 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-08-19T11:10:20.290 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-08-19T11:10:20.316 INFO:teuthology.orchestra.run.smithi130.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy InRelease 2024-08-19T11:10:20.341 INFO:teuthology.orchestra.run.smithi130.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy Release [7,005 B] 2024-08-19T11:10:20.363 INFO:teuthology.orchestra.run.smithi130.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy Release.gpg 2024-08-19T11:10:20.384 INFO:teuthology.orchestra.run.smithi130.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-08-19T11:10:20.412 INFO:teuthology.orchestra.run.smithi130.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-08-19T11:10:20.610 INFO:teuthology.orchestra.run.smithi130.stdout:Fetched 28.6 kB in 0s (84.5 kB/s) 2024-08-19T11:10:21.561 INFO:teuthology.orchestra.run.smithi130.stdout:Reading package lists... 2024-08-19T11:10:21.577 DEBUG:teuthology.orchestra.run.smithi130:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-5847-g79766939-1jammy cephadm=19.0.0-5847-g79766939-1jammy ceph-mds=19.0.0-5847-g79766939-1jammy ceph-mgr=19.0.0-5847-g79766939-1jammy ceph-common=19.0.0-5847-g79766939-1jammy ceph-fuse=19.0.0-5847-g79766939-1jammy ceph-test=19.0.0-5847-g79766939-1jammy ceph-volume=19.0.0-5847-g79766939-1jammy radosgw=19.0.0-5847-g79766939-1jammy python3-rados=19.0.0-5847-g79766939-1jammy python3-rgw=19.0.0-5847-g79766939-1jammy python3-cephfs=19.0.0-5847-g79766939-1jammy python3-rbd=19.0.0-5847-g79766939-1jammy libcephfs2=19.0.0-5847-g79766939-1jammy libcephfs-dev=19.0.0-5847-g79766939-1jammy librados2=19.0.0-5847-g79766939-1jammy librbd1=19.0.0-5847-g79766939-1jammy rbd-fuse=19.0.0-5847-g79766939-1jammy 2024-08-19T11:10:21.639 INFO:teuthology.orchestra.run.smithi130.stdout:Reading package lists... 2024-08-19T11:10:21.847 INFO:teuthology.orchestra.run.smithi130.stdout:Building dependency tree... 2024-08-19T11:10:21.848 INFO:teuthology.orchestra.run.smithi130.stdout:Reading state information... 2024-08-19T11:10:22.043 INFO:teuthology.orchestra.run.smithi130.stdout:The following packages were automatically installed and are no longer required: 2024-08-19T11:10:22.043 INFO:teuthology.orchestra.run.smithi130.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-19T11:10:22.043 INFO:teuthology.orchestra.run.smithi130.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-19T11:10:22.043 INFO:teuthology.orchestra.run.smithi130.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-19T11:10:22.043 INFO:teuthology.orchestra.run.smithi130.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-08-19T11:10:22.044 INFO:teuthology.orchestra.run.smithi130.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-08-19T11:10:22.044 INFO:teuthology.orchestra.run.smithi130.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-19T11:10:22.044 INFO:teuthology.orchestra.run.smithi130.stdout:The following additional packages will be installed: 2024-08-19T11:10:22.044 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-08-19T11:10:22.044 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-08-19T11:10:22.045 INFO:teuthology.orchestra.run.smithi130.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-08-19T11:10:22.045 INFO:teuthology.orchestra.run.smithi130.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-08-19T11:10:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-08-19T11:10:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-08-19T11:10:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-08-19T11:10:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-08-19T11:10:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-08-19T11:10:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-08-19T11:10:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-08-19T11:10:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-08-19T11:10:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-08-19T11:10:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-08-19T11:10:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-08-19T11:10:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-08-19T11:10:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-08-19T11:10:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-08-19T11:10:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-08-19T11:10:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-08-19T11:10:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-08-19T11:10:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-08-19T11:10:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-08-19T11:10:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet zip 2024-08-19T11:10:22.048 INFO:teuthology.orchestra.run.smithi130.stdout:Suggested packages: 2024-08-19T11:10:22.048 INFO:teuthology.orchestra.run.smithi130.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-08-19T11:10:22.048 INFO:teuthology.orchestra.run.smithi130.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-08-19T11:10:22.048 INFO:teuthology.orchestra.run.smithi130.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-08-19T11:10:22.048 INFO:teuthology.orchestra.run.smithi130.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-08-19T11:10:22.048 INFO:teuthology.orchestra.run.smithi130.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-08-19T11:10:22.048 INFO:teuthology.orchestra.run.smithi130.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-08-19T11:10:22.048 INFO:teuthology.orchestra.run.smithi130.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-08-19T11:10:22.048 INFO:teuthology.orchestra.run.smithi130.stdout:Recommended packages: 2024-08-19T11:10:22.048 INFO:teuthology.orchestra.run.smithi130.stdout: btrfs-tools 2024-08-19T11:10:22.148 INFO:teuthology.orchestra.run.smithi130.stdout:The following NEW packages will be installed: 2024-08-19T11:10:22.149 INFO:teuthology.orchestra.run.smithi130.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-08-19T11:10:22.149 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-08-19T11:10:22.149 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-08-19T11:10:22.149 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-08-19T11:10:22.149 INFO:teuthology.orchestra.run.smithi130.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-08-19T11:10:22.149 INFO:teuthology.orchestra.run.smithi130.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-08-19T11:10:22.151 INFO:teuthology.orchestra.run.smithi130.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-08-19T11:10:22.151 INFO:teuthology.orchestra.run.smithi130.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-08-19T11:10:22.151 INFO:teuthology.orchestra.run.smithi130.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-08-19T11:10:22.151 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-08-19T11:10:22.151 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-08-19T11:10:22.151 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-08-19T11:10:22.151 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-08-19T11:10:22.151 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-08-19T11:10:22.151 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-08-19T11:10:22.152 INFO:teuthology.orchestra.run.smithi130.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-08-19T11:10:22.152 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-08-19T11:10:22.152 INFO:teuthology.orchestra.run.smithi130.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-08-19T11:10:22.152 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-08-19T11:10:22.152 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-08-19T11:10:22.152 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-08-19T11:10:22.152 INFO:teuthology.orchestra.run.smithi130.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-08-19T11:10:22.152 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-08-19T11:10:22.152 INFO:teuthology.orchestra.run.smithi130.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-08-19T11:10:22.152 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-08-19T11:10:22.152 INFO:teuthology.orchestra.run.smithi130.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-08-19T11:10:22.295 INFO:teuthology.orchestra.run.smithi130.stdout:0 upgraded, 118 newly installed, 0 to remove and 320 not upgraded. 2024-08-19T11:10:22.296 INFO:teuthology.orchestra.run.smithi130.stdout:Need to get 251 MB of archives. 2024-08-19T11:10:22.296 INFO:teuthology.orchestra.run.smithi130.stdout:After this operation, 932 MB of additional disk space will be used. 2024-08-19T11:10:22.296 INFO:teuthology.orchestra.run.smithi130.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-5847-g79766939-1jammy [3,632 kB] 2024-08-19T11:10:22.340 INFO:teuthology.orchestra.run.smithi130.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-08-19T11:10:22.489 INFO:teuthology.orchestra.run.smithi130.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-5847-g79766939-1jammy [3,313 kB] 2024-08-19T11:10:22.535 INFO:teuthology.orchestra.run.smithi130.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-5847-g79766939-1jammy [721 kB] 2024-08-19T11:10:22.547 INFO:teuthology.orchestra.run.smithi130.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-5847-g79766939-1jammy [356 kB] 2024-08-19T11:10:22.552 INFO:teuthology.orchestra.run.smithi130.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-5847-g79766939-1jammy [32.8 kB] 2024-08-19T11:10:22.553 INFO:teuthology.orchestra.run.smithi130.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-5847-g79766939-1jammy [180 kB] 2024-08-19T11:10:22.556 INFO:teuthology.orchestra.run.smithi130.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-5847-g79766939-1jammy [68.9 kB] 2024-08-19T11:10:22.557 INFO:teuthology.orchestra.run.smithi130.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-5847-g79766939-1jammy [338 kB] 2024-08-19T11:10:22.562 INFO:teuthology.orchestra.run.smithi130.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-5847-g79766939-1jammy [8,017 kB] 2024-08-19T11:10:22.694 INFO:teuthology.orchestra.run.smithi130.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-5847-g79766939-1jammy [112 kB] 2024-08-19T11:10:22.696 INFO:teuthology.orchestra.run.smithi130.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-5847-g79766939-1jammy [472 kB] 2024-08-19T11:10:22.703 INFO:teuthology.orchestra.run.smithi130.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-5847-g79766939-1jammy [28.3 MB] 2024-08-19T11:10:22.764 INFO:teuthology.orchestra.run.smithi130.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-08-19T11:10:22.776 INFO:teuthology.orchestra.run.smithi130.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-08-19T11:10:22.858 INFO:teuthology.orchestra.run.smithi130.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-08-19T11:10:23.089 INFO:teuthology.orchestra.run.smithi130.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-08-19T11:10:23.099 INFO:teuthology.orchestra.run.smithi130.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-08-19T11:10:23.129 INFO:teuthology.orchestra.run.smithi130.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-08-19T11:10:23.138 INFO:teuthology.orchestra.run.smithi130.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-08-19T11:10:23.139 INFO:teuthology.orchestra.run.smithi130.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-08-19T11:10:23.140 INFO:teuthology.orchestra.run.smithi130.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-08-19T11:10:23.158 INFO:teuthology.orchestra.run.smithi130.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-08-19T11:10:23.163 INFO:teuthology.orchestra.run.smithi130.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-08-19T11:10:23.167 INFO:teuthology.orchestra.run.smithi130.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-08-19T11:10:23.224 INFO:teuthology.orchestra.run.smithi130.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-08-19T11:10:23.253 INFO:teuthology.orchestra.run.smithi130.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-08-19T11:10:23.281 INFO:teuthology.orchestra.run.smithi130.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-08-19T11:10:23.281 INFO:teuthology.orchestra.run.smithi130.stdout:Get:29 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-5847-g79766939-1jammy [5,198 kB] 2024-08-19T11:10:23.285 INFO:teuthology.orchestra.run.smithi130.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-08-19T11:10:23.288 INFO:teuthology.orchestra.run.smithi130.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-08-19T11:10:23.290 INFO:teuthology.orchestra.run.smithi130.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-08-19T11:10:23.290 INFO:teuthology.orchestra.run.smithi130.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-08-19T11:10:23.293 INFO:teuthology.orchestra.run.smithi130.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-08-19T11:10:23.293 INFO:teuthology.orchestra.run.smithi130.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-08-19T11:10:23.335 INFO:teuthology.orchestra.run.smithi130.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-08-19T11:10:23.335 INFO:teuthology.orchestra.run.smithi130.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-08-19T11:10:23.343 INFO:teuthology.orchestra.run.smithi130.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-5847-g79766939-1jammy [247 kB] 2024-08-19T11:10:23.346 INFO:teuthology.orchestra.run.smithi130.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-5847-g79766939-1jammy [124 kB] 2024-08-19T11:10:23.347 INFO:teuthology.orchestra.run.smithi130.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-5847-g79766939-1jammy [1,560 kB] 2024-08-19T11:10:23.361 INFO:teuthology.orchestra.run.smithi130.stdout:Get:41 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-5847-g79766939-1jammy [6,293 kB] 2024-08-19T11:10:23.416 INFO:teuthology.orchestra.run.smithi130.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-08-19T11:10:23.416 INFO:teuthology.orchestra.run.smithi130.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-08-19T11:10:23.417 INFO:teuthology.orchestra.run.smithi130.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-08-19T11:10:23.422 INFO:teuthology.orchestra.run.smithi130.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-08-19T11:10:23.424 INFO:teuthology.orchestra.run.smithi130.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-08-19T11:10:23.425 INFO:teuthology.orchestra.run.smithi130.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-08-19T11:10:23.425 INFO:teuthology.orchestra.run.smithi130.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-08-19T11:10:23.427 INFO:teuthology.orchestra.run.smithi130.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-08-19T11:10:23.454 INFO:teuthology.orchestra.run.smithi130.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-5847-g79766939-1jammy [23.1 MB] 2024-08-19T11:10:23.497 INFO:teuthology.orchestra.run.smithi130.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-08-19T11:10:23.498 INFO:teuthology.orchestra.run.smithi130.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-08-19T11:10:23.579 INFO:teuthology.orchestra.run.smithi130.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-08-19T11:10:23.581 INFO:teuthology.orchestra.run.smithi130.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-08-19T11:10:23.581 INFO:teuthology.orchestra.run.smithi130.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-08-19T11:10:23.593 INFO:teuthology.orchestra.run.smithi130.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-08-19T11:10:23.594 INFO:teuthology.orchestra.run.smithi130.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-08-19T11:10:23.595 INFO:teuthology.orchestra.run.smithi130.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-08-19T11:10:23.596 INFO:teuthology.orchestra.run.smithi130.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-08-19T11:10:23.599 INFO:teuthology.orchestra.run.smithi130.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-08-19T11:10:23.661 INFO:teuthology.orchestra.run.smithi130.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-08-19T11:10:23.662 INFO:teuthology.orchestra.run.smithi130.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-08-19T11:10:23.685 INFO:teuthology.orchestra.run.smithi130.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-08-19T11:10:23.687 INFO:teuthology.orchestra.run.smithi130.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-08-19T11:10:23.692 INFO:teuthology.orchestra.run.smithi130.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-08-19T11:10:23.693 INFO:teuthology.orchestra.run.smithi130.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-08-19T11:10:23.743 INFO:teuthology.orchestra.run.smithi130.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-08-19T11:10:23.748 INFO:teuthology.orchestra.run.smithi130.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-08-19T11:10:23.892 INFO:teuthology.orchestra.run.smithi130.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-5847-g79766939-1jammy [14.1 kB] 2024-08-19T11:10:23.957 INFO:teuthology.orchestra.run.smithi130.stdout:Get:70 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-5847-g79766939-1jammy [880 kB] 2024-08-19T11:10:23.958 INFO:teuthology.orchestra.run.smithi130.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-08-19T11:10:23.959 INFO:teuthology.orchestra.run.smithi130.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-08-19T11:10:23.962 INFO:teuthology.orchestra.run.smithi130.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-08-19T11:10:23.963 INFO:teuthology.orchestra.run.smithi130.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-08-19T11:10:23.965 INFO:teuthology.orchestra.run.smithi130.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-08-19T11:10:23.969 INFO:teuthology.orchestra.run.smithi130.stdout:Get:76 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-5847-g79766939-1jammy [2,437 kB] 2024-08-19T11:10:24.000 INFO:teuthology.orchestra.run.smithi130.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-5847-g79766939-1jammy [801 kB] 2024-08-19T11:10:24.007 INFO:teuthology.orchestra.run.smithi130.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-08-19T11:10:24.009 INFO:teuthology.orchestra.run.smithi130.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-5847-g79766939-1jammy [156 kB] 2024-08-19T11:10:24.011 INFO:teuthology.orchestra.run.smithi130.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-08-19T11:10:24.011 INFO:teuthology.orchestra.run.smithi130.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-5847-g79766939-1jammy [61.3 MB] 2024-08-19T11:10:24.012 INFO:teuthology.orchestra.run.smithi130.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-08-19T11:10:24.078 INFO:teuthology.orchestra.run.smithi130.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-08-19T11:10:24.078 INFO:teuthology.orchestra.run.smithi130.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-08-19T11:10:24.128 INFO:teuthology.orchestra.run.smithi130.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-08-19T11:10:24.129 INFO:teuthology.orchestra.run.smithi130.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-08-19T11:10:24.130 INFO:teuthology.orchestra.run.smithi130.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-08-19T11:10:24.131 INFO:teuthology.orchestra.run.smithi130.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-08-19T11:10:24.132 INFO:teuthology.orchestra.run.smithi130.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-08-19T11:10:24.133 INFO:teuthology.orchestra.run.smithi130.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-08-19T11:10:24.141 INFO:teuthology.orchestra.run.smithi130.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-08-19T11:10:24.214 INFO:teuthology.orchestra.run.smithi130.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-08-19T11:10:24.218 INFO:teuthology.orchestra.run.smithi130.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-08-19T11:10:24.219 INFO:teuthology.orchestra.run.smithi130.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-08-19T11:10:24.298 INFO:teuthology.orchestra.run.smithi130.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-08-19T11:10:24.305 INFO:teuthology.orchestra.run.smithi130.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-08-19T11:10:24.307 INFO:teuthology.orchestra.run.smithi130.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-08-19T11:10:24.309 INFO:teuthology.orchestra.run.smithi130.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-08-19T11:10:24.320 INFO:teuthology.orchestra.run.smithi130.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-08-19T11:10:24.322 INFO:teuthology.orchestra.run.smithi130.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-08-19T11:10:24.330 INFO:teuthology.orchestra.run.smithi130.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-08-19T11:10:24.380 INFO:teuthology.orchestra.run.smithi130.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-08-19T11:10:24.382 INFO:teuthology.orchestra.run.smithi130.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-08-19T11:10:24.382 INFO:teuthology.orchestra.run.smithi130.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-08-19T11:10:24.462 INFO:teuthology.orchestra.run.smithi130.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-08-19T11:10:24.464 INFO:teuthology.orchestra.run.smithi130.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-08-19T11:10:24.482 INFO:teuthology.orchestra.run.smithi130.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-08-19T11:10:24.483 INFO:teuthology.orchestra.run.smithi130.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-08-19T11:10:24.484 INFO:teuthology.orchestra.run.smithi130.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-08-19T11:10:24.489 INFO:teuthology.orchestra.run.smithi130.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-08-19T11:10:24.491 INFO:teuthology.orchestra.run.smithi130.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-08-19T11:10:25.305 INFO:teuthology.orchestra.run.smithi130.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-5847-g79766939-1jammy [8,625 kB] 2024-08-19T11:10:25.413 INFO:teuthology.orchestra.run.smithi130.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-5847-g79766939-1jammy [14.2 kB] 2024-08-19T11:10:25.414 INFO:teuthology.orchestra.run.smithi130.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-5847-g79766939-1jammy [55.8 MB] 2024-08-19T11:10:26.596 INFO:teuthology.orchestra.run.smithi130.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-5847-g79766939-1jammy [130 kB] 2024-08-19T11:10:26.598 INFO:teuthology.orchestra.run.smithi130.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-5847-g79766939-1jammy [38.9 kB] 2024-08-19T11:10:26.600 INFO:teuthology.orchestra.run.smithi130.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-5847-g79766939-1jammy [16.0 MB] 2024-08-19T11:10:26.908 INFO:teuthology.orchestra.run.smithi130.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-5847-g79766939-1jammy [91.4 kB] 2024-08-19T11:10:27.477 INFO:teuthology.orchestra.run.smithi130.stdout:Fetched 251 MB in 5s (52.8 MB/s) 2024-08-19T11:10:27.601 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-08-19T11:10:27.696 INFO:teuthology.orchestra.run.smithi130.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-08-19T11:10:27.701 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-08-19T11:10:27.775 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-08-19T11:10:28.111 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-08-19T11:10:28.137 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-08-19T11:10:28.178 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-08-19T11:10:28.631 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-08-19T11:10:28.657 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-08-19T11:10:28.697 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-08-19T11:10:29.117 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-08-19T11:10:29.143 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-19T11:10:29.193 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-19T11:10:29.512 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-08-19T11:10:29.538 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-19T11:10:29.587 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-19T11:10:29.923 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-08-19T11:10:29.949 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-19T11:10:29.989 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-19T11:10:30.342 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-08-19T11:10:30.368 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-08-19T11:10:30.408 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-08-19T11:10:30.703 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package librados2. 2024-08-19T11:10:30.729 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../007-librados2_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:30.769 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking librados2 (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:31.156 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package librbd1. 2024-08-19T11:10:31.182 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../008-librbd1_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:31.222 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking librbd1 (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:31.675 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libcephfs2. 2024-08-19T11:10:31.701 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:31.741 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libcephfs2 (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:32.019 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-rados. 2024-08-19T11:10:32.034 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../010-python3-rados_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:32.077 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-rados (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:32.404 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-08-19T11:10:32.431 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-19T11:10:32.471 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-ceph-argparse (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:32.673 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-cephfs. 2024-08-19T11:10:32.688 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:32.723 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-cephfs (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:33.034 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-ceph-common. 2024-08-19T11:10:33.060 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-19T11:10:33.101 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-ceph-common (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:33.420 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-wcwidth. 2024-08-19T11:10:33.446 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-08-19T11:10:33.487 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-08-19T11:10:33.915 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-prettytable. 2024-08-19T11:10:33.941 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-08-19T11:10:33.981 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-08-19T11:10:34.217 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-rbd. 2024-08-19T11:10:34.243 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:34.275 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-rbd (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:34.577 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-08-19T11:10:34.604 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-08-19T11:10:34.644 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-08-19T11:10:34.922 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-08-19T11:10:34.937 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-08-19T11:10:34.980 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-08-19T11:10:35.282 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-08-19T11:10:35.297 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-08-19T11:10:35.341 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-08-19T11:10:35.627 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package lua5.1. 2024-08-19T11:10:35.654 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-08-19T11:10:35.685 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-08-19T11:10:35.997 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package lua-any. 2024-08-19T11:10:36.023 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-08-19T11:10:36.063 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking lua-any (27ubuntu1) ... 2024-08-19T11:10:36.341 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package zip. 2024-08-19T11:10:36.367 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-08-19T11:10:36.407 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking zip (3.0-12build2) ... 2024-08-19T11:10:36.702 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package unzip. 2024-08-19T11:10:36.728 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-08-19T11:10:36.768 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-08-19T11:10:37.166 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package luarocks. 2024-08-19T11:10:37.192 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-08-19T11:10:37.232 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-08-19T11:10:37.615 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package librgw2. 2024-08-19T11:10:37.641 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../025-librgw2_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:37.681 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking librgw2 (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:38.292 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-rgw. 2024-08-19T11:10:38.319 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:38.359 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-rgw (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:38.637 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package liboath0:amd64. 2024-08-19T11:10:38.664 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-08-19T11:10:38.703 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-08-19T11:10:39.023 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libradosstriper1. 2024-08-19T11:10:39.050 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:39.090 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libradosstriper1 (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:39.493 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-common. 2024-08-19T11:10:39.519 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../029-ceph-common_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:39.560 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-common (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:40.981 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-base. 2024-08-19T11:10:41.007 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../030-ceph-base_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:41.057 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-base (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:41.619 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-08-19T11:10:41.645 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-08-19T11:10:41.685 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-08-19T11:10:41.988 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-cheroot. 2024-08-19T11:10:42.014 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-08-19T11:10:42.054 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-08-19T11:10:42.366 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-08-19T11:10:42.392 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-08-19T11:10:42.433 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-08-19T11:10:42.677 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-jaraco.text. 2024-08-19T11:10:42.703 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-08-19T11:10:42.743 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-08-19T11:10:43.046 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-08-19T11:10:43.073 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-08-19T11:10:43.113 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-08-19T11:10:43.416 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-tempora. 2024-08-19T11:10:43.442 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-08-19T11:10:43.482 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-08-19T11:10:43.744 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-portend. 2024-08-19T11:10:43.770 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-08-19T11:10:43.802 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-08-19T11:10:44.096 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-08-19T11:10:44.122 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-08-19T11:10:44.163 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-08-19T11:10:44.474 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-cherrypy3. 2024-08-19T11:10:44.501 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-08-19T11:10:44.541 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-08-19T11:10:44.853 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-natsort. 2024-08-19T11:10:44.879 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-08-19T11:10:44.919 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-08-19T11:10:45.230 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-logutils. 2024-08-19T11:10:45.257 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-08-19T11:10:45.297 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-08-19T11:10:45.600 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-markupsafe. 2024-08-19T11:10:45.626 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-08-19T11:10:45.666 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-08-19T11:10:45.928 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-mako. 2024-08-19T11:10:45.943 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-08-19T11:10:45.977 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-08-19T11:10:46.289 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-simplegeneric. 2024-08-19T11:10:46.315 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-08-19T11:10:46.356 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-08-19T11:10:46.659 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-singledispatch. 2024-08-19T11:10:46.685 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-08-19T11:10:46.725 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-08-19T11:10:47.019 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-webob. 2024-08-19T11:10:47.046 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-08-19T11:10:47.086 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-08-19T11:10:47.406 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-waitress. 2024-08-19T11:10:47.432 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-08-19T11:10:47.478 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-08-19T11:10:47.775 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-tempita. 2024-08-19T11:10:47.802 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-08-19T11:10:47.843 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-08-19T11:10:48.161 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-paste. 2024-08-19T11:10:48.188 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-08-19T11:10:48.228 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-08-19T11:10:48.514 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-08-19T11:10:48.530 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-08-19T11:10:48.564 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-08-19T11:10:48.851 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pastedeploy. 2024-08-19T11:10:48.877 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-08-19T11:10:48.917 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-08-19T11:10:49.220 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-webtest. 2024-08-19T11:10:49.247 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-08-19T11:10:49.287 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-08-19T11:10:49.581 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pecan. 2024-08-19T11:10:49.607 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-08-19T11:10:49.648 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-08-19T11:10:49.977 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-certifi. 2024-08-19T11:10:50.004 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-08-19T11:10:50.036 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-08-19T11:10:50.331 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-idna. 2024-08-19T11:10:50.357 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-08-19T11:10:50.397 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-08-19T11:10:50.716 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-urllib3. 2024-08-19T11:10:50.743 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-08-19T11:10:50.783 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-08-19T11:10:51.094 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-requests. 2024-08-19T11:10:51.121 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-08-19T11:10:51.161 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-08-19T11:10:51.472 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-werkzeug. 2024-08-19T11:10:51.499 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-08-19T11:10:51.538 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-08-19T11:10:51.850 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-08-19T11:10:51.866 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-19T11:10:51.908 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mgr-modules-core (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:52.288 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-08-19T11:10:52.315 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:52.354 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:52.674 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-bcrypt. 2024-08-19T11:10:52.701 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-08-19T11:10:52.740 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-08-19T11:10:53.002 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-openssl. 2024-08-19T11:10:53.029 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-08-19T11:10:53.068 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-08-19T11:10:53.371 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mgr. 2024-08-19T11:10:53.398 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:53.438 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mgr (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:53.849 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mon. 2024-08-19T11:10:53.876 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:53.916 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mon (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:54.519 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-osd. 2024-08-19T11:10:54.546 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:54.586 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-osd (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:55.714 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph. 2024-08-19T11:10:55.741 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../066-ceph_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:56.106 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:56.759 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-fuse. 2024-08-19T11:10:56.786 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:56.825 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-fuse (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:57.204 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mds. 2024-08-19T11:10:57.230 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:57.270 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mds (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:57.623 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package cephadm. 2024-08-19T11:10:57.650 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../069-cephadm_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:10:57.681 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking cephadm (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:10:57.992 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-asyncssh. 2024-08-19T11:10:58.019 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-08-19T11:10:58.058 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-08-19T11:10:58.328 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python-babel-localedata. 2024-08-19T11:10:58.344 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-08-19T11:10:58.386 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-08-19T11:10:59.717 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-babel. 2024-08-19T11:10:59.744 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-08-19T11:10:59.783 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-08-19T11:11:00.086 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-jinja2. 2024-08-19T11:11:00.112 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-08-19T11:11:00.144 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-08-19T11:11:00.447 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-08-19T11:11:00.462 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-19T11:11:00.505 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mgr-cephadm (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:00.817 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-repoze.lru. 2024-08-19T11:11:00.844 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-08-19T11:11:00.884 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-08-19T11:11:01.145 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-routes. 2024-08-19T11:11:01.172 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-08-19T11:11:01.203 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-08-19T11:11:01.556 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-08-19T11:11:01.583 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-19T11:11:01.623 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mgr-dashboard (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:04.328 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-08-19T11:11:04.357 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-08-19T11:11:04.385 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-08-19T11:11:04.831 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-joblib. 2024-08-19T11:11:04.859 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-08-19T11:11:04.897 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-08-19T11:11:05.284 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-08-19T11:11:05.312 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-08-19T11:11:05.350 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-08-19T11:11:05.637 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-sklearn. 2024-08-19T11:11:05.664 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-08-19T11:11:05.703 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-08-19T11:11:06.287 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-08-19T11:11:06.306 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-19T11:11:06.422 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:07.409 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-cachetools. 2024-08-19T11:11:07.436 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-08-19T11:11:07.475 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-08-19T11:11:07.739 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pyasn1. 2024-08-19T11:11:07.764 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-08-19T11:11:07.803 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-08-19T11:11:08.089 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-08-19T11:11:08.105 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-08-19T11:11:08.139 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-08-19T11:11:08.393 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-rsa. 2024-08-19T11:11:08.420 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-08-19T11:11:08.451 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-rsa (4.8-1) ... 2024-08-19T11:11:08.771 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-google-auth. 2024-08-19T11:11:08.799 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-08-19T11:11:08.837 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-08-19T11:11:09.141 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-08-19T11:11:09.169 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-08-19T11:11:09.216 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-08-19T11:11:09.527 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-websocket. 2024-08-19T11:11:09.555 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-08-19T11:11:09.594 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-08-19T11:11:09.906 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-kubernetes. 2024-08-19T11:11:09.933 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-08-19T11:11:09.972 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-08-19T11:11:10.611 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-08-19T11:11:10.627 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-19T11:11:10.669 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:11.014 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libonig5:amd64. 2024-08-19T11:11:11.042 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-08-19T11:11:11.080 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-08-19T11:11:11.418 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libjq1:amd64. 2024-08-19T11:11:11.444 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-08-19T11:11:11.491 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-08-19T11:11:11.735 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package jq. 2024-08-19T11:11:11.751 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-08-19T11:11:11.785 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-08-19T11:11:11.988 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package socat. 2024-08-19T11:11:12.004 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-08-19T11:11:12.046 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-08-19T11:11:12.349 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package xmlstarlet. 2024-08-19T11:11:12.365 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-08-19T11:11:12.407 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-08-19T11:11:12.737 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-test. 2024-08-19T11:11:12.753 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../097-ceph-test_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:11:12.786 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-test (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:15.450 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-volume. 2024-08-19T11:11:15.586 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-19T11:11:15.586 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-volume (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:15.787 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libcephfs-dev. 2024-08-19T11:11:15.815 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:11:15.853 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libcephfs-dev (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:16.164 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package lua-socket:amd64. 2024-08-19T11:11:16.193 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-08-19T11:11:16.231 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-08-19T11:11:16.561 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package lua-sec:amd64. 2024-08-19T11:11:16.588 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-08-19T11:11:16.627 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-08-19T11:11:16.989 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package nvme-cli. 2024-08-19T11:11:17.017 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-08-19T11:11:17.056 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-08-19T11:11:17.446 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package pkg-config. 2024-08-19T11:11:17.475 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-08-19T11:11:17.513 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-08-19T11:11:17.765 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-08-19T11:11:17.779 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-08-19T11:11:17.824 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-08-19T11:11:18.253 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-iniconfig. 2024-08-19T11:11:18.280 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-08-19T11:11:18.318 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-08-19T11:11:18.647 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pastescript. 2024-08-19T11:11:18.675 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-08-19T11:11:18.713 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-08-19T11:11:19.017 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pluggy. 2024-08-19T11:11:19.044 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-08-19T11:11:19.083 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-08-19T11:11:19.378 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-psutil. 2024-08-19T11:11:19.406 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-08-19T11:11:19.444 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-08-19T11:11:19.764 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-py. 2024-08-19T11:11:19.792 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-08-19T11:11:19.830 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-py (1.10.0-1) ... 2024-08-19T11:11:20.125 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pygments. 2024-08-19T11:11:20.153 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-08-19T11:11:20.192 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-08-19T11:11:20.655 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pyinotify. 2024-08-19T11:11:20.683 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-08-19T11:11:20.721 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-08-19T11:11:21.032 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-toml. 2024-08-19T11:11:21.061 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-08-19T11:11:21.098 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-08-19T11:11:21.360 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pytest. 2024-08-19T11:11:21.376 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-08-19T11:11:21.418 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-08-19T11:11:21.763 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-simplejson. 2024-08-19T11:11:21.792 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-08-19T11:11:21.830 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-08-19T11:11:22.107 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-08-19T11:11:22.124 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-08-19T11:11:22.166 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-08-19T11:11:22.895 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package radosgw. 2024-08-19T11:11:22.923 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../116-radosgw_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:11:22.961 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking radosgw (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:23.848 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package rbd-fuse. 2024-08-19T11:11:23.861 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-19T11:11:23.906 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking rbd-fuse (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:24.194 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-08-19T11:11:24.470 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-08-19T11:11:24.587 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-08-19T11:11:24.843 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-08-19T11:11:25.142 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-08-19T11:11:25.528 INFO:teuthology.orchestra.run.smithi130.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-08-19T11:11:25.865 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up cephadm (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:26.357 INFO:teuthology.orchestra.run.smithi130.stdout:Adding system user cephadm....done 2024-08-19T11:11:26.513 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-08-19T11:11:26.823 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-08-19T11:11:27.074 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-08-19T11:11:27.200 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-08-19T11:11:27.451 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-08-19T11:11:27.720 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-08-19T11:11:27.846 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-py (1.10.0-1) ... 2024-08-19T11:11:28.147 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-08-19T11:11:28.507 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-08-19T11:11:28.767 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-08-19T11:11:29.035 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-08-19T11:11:29.278 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-08-19T11:11:29.521 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-08-19T11:11:29.763 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-08-19T11:11:30.106 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-ceph-argparse (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:30.374 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-08-19T11:11:30.500 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-08-19T11:11:30.609 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-08-19T11:11:30.719 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-08-19T11:11:30.970 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-08-19T11:11:31.350 INFO:teuthology.orchestra.run.smithi130.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-08-19T11:11:31.384 INFO:teuthology.orchestra.run.smithi130.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-08-19T11:11:31.457 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-08-19T11:11:31.583 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-08-19T11:11:31.942 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-08-19T11:11:32.051 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-08-19T11:11:32.311 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-08-19T11:11:32.562 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-08-19T11:11:32.855 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up zip (3.0-12build2) ... 2024-08-19T11:11:32.981 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-08-19T11:11:33.633 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-08-19T11:11:33.893 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-08-19T11:11:34.035 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-08-19T11:11:34.161 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-08-19T11:11:34.471 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-08-19T11:11:34.839 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-08-19T11:11:35.090 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-08-19T11:11:35.475 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-08-19T11:11:35.759 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-08-19T11:11:36.103 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-08-19T11:11:36.345 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-08-19T11:11:36.471 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-ceph-common (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:36.773 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-08-19T11:11:37.115 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-08-19T11:11:38.330 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-08-19T11:11:38.581 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-19T11:11:38.716 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-toml (0.10.2-1) ... 2024-08-19T11:11:38.983 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-08-19T11:11:39.109 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-08-19T11:11:39.261 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-08-19T11:11:39.554 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-08-19T11:11:39.805 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-08-19T11:11:40.064 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-19T11:11:40.190 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-08-19T11:11:40.483 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-08-19T11:11:40.718 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-08-19T11:11:40.986 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-08-19T11:11:41.237 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-08-19T11:11:41.505 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-08-19T11:11:41.764 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-08-19T11:11:41.890 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-08-19T11:11:42.166 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-08-19T11:11:42.276 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-08-19T11:11:42.552 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-08-19T11:11:42.845 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-08-19T11:11:43.088 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-08-19T11:11:43.197 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-08-19T11:11:43.323 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-08-19T11:11:43.412 INFO:teuthology.orchestra.run.smithi130.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-08-19T11:11:43.650 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-08-19T11:11:43.759 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-08-19T11:11:44.127 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-08-19T11:11:44.387 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up lua-any (27ubuntu1) ... 2024-08-19T11:11:44.513 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-portend (3.0.0-1) ... 2024-08-19T11:11:44.764 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-19T11:11:44.890 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-08-19T11:11:45.224 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-08-19T11:11:45.542 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-08-19T11:11:45.848 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-08-19T11:11:46.365 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-rsa (4.8-1) ... 2024-08-19T11:11:46.621 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-08-19T11:11:46.872 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-08-19T11:11:47.225 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-08-19T11:11:47.610 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-08-19T11:11:47.895 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-08-19T11:11:48.231 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-08-19T11:11:48.357 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up librados2 (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:48.483 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libsqlite3-mod-ceph (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:48.609 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-08-19T11:11:48.921 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libcephfs2 (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:49.030 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libradosstriper1 (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:49.181 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-08-19T11:11:49.466 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up librbd1 (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:49.592 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mgr-modules-core (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:49.718 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-fuse (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:49.937 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-08-19T11:11:49.937 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-08-19T11:11:50.395 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libcephfs-dev (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:50.513 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-rados (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:50.622 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up librgw2 (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:50.748 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-08-19T11:11:51.952 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-rbd (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:52.078 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up rbd-fuse (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:52.204 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-rgw (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:52.346 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-cephfs (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:52.473 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-common (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:52.974 INFO:teuthology.orchestra.run.smithi130.stdout:Adding group ceph....done 2024-08-19T11:11:53.267 INFO:teuthology.orchestra.run.smithi130.stdout:Adding system user ceph....done 2024-08-19T11:11:53.428 INFO:teuthology.orchestra.run.smithi130.stdout:Setting system user ceph properties....done 2024-08-19T11:11:53.445 INFO:teuthology.orchestra.run.smithi130.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-08-19T11:11:53.585 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-08-19T11:11:53.921 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-08-19T11:11:54.383 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-test (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:54.492 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up radosgw (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:54.960 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-08-19T11:11:54.961 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-08-19T11:11:55.427 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-base (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:55.781 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-08-19T11:11:56.271 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mds (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:56.495 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-08-19T11:11:56.495 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-08-19T11:11:56.973 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mgr (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:57.190 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-08-19T11:11:57.190 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-08-19T11:11:57.657 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-osd (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:57.932 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-08-19T11:11:57.932 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-08-19T11:11:58.417 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mgr-k8sevents (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:58.552 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:58.703 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mon (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:58.915 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-08-19T11:11:58.915 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-08-19T11:11:59.371 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mgr-cephadm (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:59.505 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:59.631 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mgr-dashboard (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:11:59.774 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-volume (19.0.0-5847-g79766939-1jammy) ... 2024-08-19T11:12:00.119 INFO:teuthology.orchestra.run.smithi130.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-08-19T11:12:00.228 INFO:teuthology.orchestra.run.smithi130.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-08-19T11:12:00.372 INFO:teuthology.orchestra.run.smithi130.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-08-19T11:12:01.448 INFO:teuthology.orchestra.run.smithi130.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-08-19T11:12:02.146 INFO:teuthology.orchestra.run.smithi130.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-08-19T11:12:02.156 DEBUG:teuthology.parallel:result is None 2024-08-19T11:12:02.156 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-19T11:12:02.308 DEBUG:teuthology.orchestra.run.smithi130:> dpkg-query -W -f '${Version}' ceph 2024-08-19T11:12:02.327 INFO:teuthology.orchestra.run.smithi130.stdout:19.0.0-5847-g79766939-1jammy 2024-08-19T11:12:02.327 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5847-g79766939-1jammy 2024-08-19T11:12:02.328 INFO:teuthology.task.install:The correct ceph version 19.0.0-5847-g79766939-1jammy is installed. 2024-08-19T11:12:02.329 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-19T11:12:02.329 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-08-19T11:12:02.330 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-19T11:12:02.382 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-19T11:12:02.382 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-08-19T11:12:02.382 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/daemon-helper 2024-08-19T11:12:02.438 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-19T11:12:02.493 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-19T11:12:02.493 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-08-19T11:12:02.493 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-19T11:12:02.546 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-19T11:12:02.601 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-19T11:12:02.601 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-08-19T11:12:02.601 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/stdin-killer 2024-08-19T11:12:02.655 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-19T11:12:02.709 INFO:teuthology.run_tasks:Running task workunit... 2024-08-19T11:12:02.727 INFO:tasks.workunit:Pulling workunits from ref 964adda23da9f7b057cb34fdc0b82aeb7ed65324 2024-08-19T11:12:02.728 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-19T11:12:02.728 INFO:tasks.workunit:timeout=3h 2024-08-19T11:12:02.728 INFO:tasks.workunit:cleanup=True 2024-08-19T11:12:02.729 DEBUG:teuthology.orchestra.run.smithi130:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-19T11:12:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T11:12:02.755 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-19T11:12:02.755 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-19T11:12:02.802 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-19T11:12:02.802 DEBUG:teuthology.orchestra.run.smithi130:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-19T11:12:02.850 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 964adda23da9f7b057cb34fdc0b82aeb7ed65324 2024-08-19T11:12:02.900 INFO:tasks.workunit.client.0.smithi130.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-19T11:12:54.738 INFO:tasks.workunit.client.0.smithi130.stderr:Note: switching to '964adda23da9f7b057cb34fdc0b82aeb7ed65324'. 2024-08-19T11:12:54.738 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-08-19T11:12:54.739 INFO:tasks.workunit.client.0.smithi130.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-19T11:12:54.739 INFO:tasks.workunit.client.0.smithi130.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-19T11:12:54.739 INFO:tasks.workunit.client.0.smithi130.stderr:state without impacting any branches by switching back to a branch. 2024-08-19T11:12:54.739 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-08-19T11:12:54.739 INFO:tasks.workunit.client.0.smithi130.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-19T11:12:54.739 INFO:tasks.workunit.client.0.smithi130.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-19T11:12:54.739 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-08-19T11:12:54.739 INFO:tasks.workunit.client.0.smithi130.stderr: git switch -c 2024-08-19T11:12:54.739 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-08-19T11:12:54.739 INFO:tasks.workunit.client.0.smithi130.stderr:Or undo this operation with: 2024-08-19T11:12:54.739 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-08-19T11:12:54.739 INFO:tasks.workunit.client.0.smithi130.stderr: git switch - 2024-08-19T11:12:54.739 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-08-19T11:12:54.740 INFO:tasks.workunit.client.0.smithi130.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-19T11:12:54.740 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-08-19T11:12:54.740 INFO:tasks.workunit.client.0.smithi130.stderr:HEAD is now at 964adda23da Merge pull request #59290 from anthonyeleven/mountpoint 2024-08-19T11:12:54.740 DEBUG:teuthology.orchestra.run.smithi130:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-19T11:12:54.788 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-08-19T11:12:54.788 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-19T11:12:54.833 INFO:tasks.workunit:Running workunits matching misc on client.0... 2024-08-19T11:12:54.834 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2024-08-19T11:12:54.834 DEBUG:teuthology.orchestra.run.smithi130:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2024-08-19T11:12:54.900 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-19T11:12:54.906 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-19T11:12:54.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-19T11:12:54.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-19T11:12:54.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export PYTHONWARNINGS=ignore 2024-08-19T11:12:54.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: PYTHONWARNINGS=ignore 2024-08-19T11:12:54.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: export CEPH_CONF=/dev/null 2024-08-19T11:12:54.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: CEPH_CONF=/dev/null 2024-08-19T11:12:54.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: unset CEPH_ARGS 2024-08-19T11:12:54.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: local code 2024-08-19T11:12:54.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: run td/mclock-config 2024-08-19T11:12:54.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2024-08-19T11:12:54.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2024-08-19T11:12:54.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-08-19T11:12:54.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-08-19T11:12:54.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2024-08-19T11:12:54.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2024-08-19T11:12:54.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none ' 2024-08-19T11:12:54.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-08-19T11:12:54.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2024-08-19T11:12:54.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2024-08-19T11:12:54.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-19T11:12:54.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2024-08-19T11:12:54.909 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_profile_builtin_to_custom 2024-08-19T11:12:54.909 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_profile_custom_to_builtin 2024-08-19T11:12:54.909 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_profile_disallow_builtin_params_modify 2024-08-19T11:12:54.910 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_profile_disallow_builtin_params_override 2024-08-19T11:12:54.910 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_recovery_limit_adjustment_mclock' 2024-08-19T11:12:54.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-08-19T11:12:54.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-08-19T11:12:54.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-08-19T11:12:54.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-08-19T11:12:54.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:12:54.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:12:54.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:12:54.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:12:54.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:12:54.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:12:54.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:12:54.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:12:54.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:12:54.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:12:54.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:12:54.912 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:12:54.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:12:54.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:12:54.913 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:12:54.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:12:54.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:12:54.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:12:54.914 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:12:54.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:12:54.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:12:54.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:12:54.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:12:54.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:12:54.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:12:54.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:12:54.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:12:54.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:12:54.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-08-19T11:12:54.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:12:54.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:12:54.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:12:54.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21434 2024-08-19T11:12:54.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:12:54.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-19T11:12:54.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-19T11:12:54.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:12:54.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-08-19T11:12:54.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2024-08-19T11:12:54.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2024-08-19T11:12:54.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2024-08-19T11:12:54.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-08-19T11:12:54.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-08-19T11:12:54.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:12:54.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:12:54.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:12:54.923 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:12:54.923 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:12:54.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:12:54.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:12:54.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:12:54.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:12:54.923 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:12:54.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:12:54.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:12:54.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:12:54.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:12:54.926 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:12:54.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:12:54.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:12:54.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:12:54.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:12:54.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:12:54.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:12:54.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:12:54.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:12:54.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:12:54.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:12:54.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:12:54.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:12:54.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:12:54.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-08-19T11:12:54.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:12:54.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:12:54.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:12:54.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21434 2024-08-19T11:12:54.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:12:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:12:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:12:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-08-19T11:12:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2024-08-19T11:12:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-08-19T11:12:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:12:54.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:12:54.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:12:54.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-08-19T11:12:54.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-08-19T11:12:55.594 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:12:55.595 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:12:55.595 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:12:55.595 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:12:55.595 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:12:55.595 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:12:55.596 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:12:55.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:12:55.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:12:55.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:12:55.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:12:55.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:12:55.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:12:55.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:12:55.783 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:12:55.783 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:12:55.783 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:12:55.784 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:12:55.784 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:12:55.784 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:12:55.784 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-mon.a.asok 2024-08-19T11:12:55.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:12:55.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21434/ceph-mon.a.asok config get fsid 2024-08-19T11:12:55.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:12:55.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:12:55.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:12:55.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:12:55.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:12:55.896 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:12:55.896 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:12:55.896 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:12:55.897 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:12:55.897 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:12:55.897 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:12:55.897 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-mon.a.asok 2024-08-19T11:12:55.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:12:55.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21434/ceph-mon.a.asok config get mon_host 2024-08-19T11:12:55.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-08-19T11:12:55.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-08-19T11:12:55.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:12:55.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:12:55.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:12:55.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-08-19T11:12:55.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:12:56.903 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:12:56.903 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:12:56.903 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:12:56.904 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:12:56.904 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:12:56.904 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:12:56.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:12:56.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:12:56.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:12:57.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-08-19T11:12:57.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-08-19T11:12:57.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:12:57.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:12:57.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:12:57.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-08-19T11:12:57.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-08-19T11:12:57.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:12:57.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:12:57.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:12:57.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-08-19T11:12:57.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-08-19T11:12:57.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:12:57.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:12:57.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-08-19T11:12:57.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:12:57.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:12:57.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:12:57.049 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:12:57.050 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:12:57.050 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:12:57.051 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:12:57.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:12:57.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:12:57.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:12:57.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:12:57.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-08-19T11:12:57.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-08-19T11:12:57.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:12:57.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:12:57.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:12:57.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:12:57.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:12:57.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-08-19T11:12:57.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-08-19T11:12:57.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:12:57.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=929108ec-bc3e-43f0-9730-8a494c85d7a3 2024-08-19T11:12:57.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 929108ec-bc3e-43f0-9730-8a494c85d7a3' 2024-08-19T11:12:57.053 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 929108ec-bc3e-43f0-9730-8a494c85d7a3 2024-08-19T11:12:57.053 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:12:57.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC5KMNmS6xGBBAAFzHSep3kypGYmnnfVdCt+A== 2024-08-19T11:12:57.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC5KMNmS6xGBBAAFzHSep3kypGYmnnfVdCt+A=="}' 2024-08-19T11:12:57.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 929108ec-bc3e-43f0-9730-8a494c85d7a3 -i td/mclock-config/0/new.json 2024-08-19T11:12:57.361 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:12:57.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-08-19T11:12:57.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQC5KMNmS6xGBBAAFzHSep3kypGYmnnfVdCt+A== --osd-uuid 929108ec-bc3e-43f0-9730-8a494c85d7a3 2024-08-19T11:12:57.433 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:12:57.428+0000 7f22f3dab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:12:57.435 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:12:57.432+0000 7f22f3dab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:12:57.436 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:12:57.432+0000 7f22f3dab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:12:57.436 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:12:57.432+0000 7f22f3dab7c0 -1 bdev(0x55f72a4bfc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:12:57.437 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:12:57.432+0000 7f22f3dab7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-08-19T11:13:03.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-08-19T11:13:03.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:13:03.505 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:13:03.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:13:03.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:13:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:13:03.754 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:13:03.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-08-19T11:13:03.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:13:03.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:13:03.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:13:03.801 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:03.800+0000 7f8ef3b767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:13:03.844 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:03.840+0000 7f8ef3b767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:13:03.846 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:03.844+0000 7f8ef3b767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:13:03.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:13:03.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:13:03.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:13:03.982 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:13:03.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:13:03.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:13:03.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:03.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:13:03.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:03.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:04.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:05.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:05.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:05.213 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:13:05.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:13:05.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:05.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:05.398 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:05.396+0000 7f8ef3b767c0 -1 Falling back to public interface 2024-08-19T11:13:05.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:06.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:06.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:06.452 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:13:06.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:13:06.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:06.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:06.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:07.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:07.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:07.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:13:07.871 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:13:07.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:07.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:08.090 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:08.088+0000 7f8ef3b767c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:13:08.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:09.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:09.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:09.478 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:13:09.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:13:09.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:09.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:09.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:10.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:10.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:10.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:13:10.929 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:13:10.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:10.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:11.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:12.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:12.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:12.358 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:13:12.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:13:12.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:12.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:12.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:13.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:13.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:13.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:13:13.898 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:13:13.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:13.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:14.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:15.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:15.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:15.357 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:13:15.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:13:15.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:15.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:15.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:16.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:16.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:16.869 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:13:16.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:13:16.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:16.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:17.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:18.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:18.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:18.304 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:13:18.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:13:18.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:18.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:18.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:19.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:19.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:19.777 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:13:19.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:13:19.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:19.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:20.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:21.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:21.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:21.674 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:13:21.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:13:21.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:21.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:22.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:23.191 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:13:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:13:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:23.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:24.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:24.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:24.898 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:13:24.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:13:24.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:24.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:25.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:26.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:26.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:26.853 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:13:26.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:13:26.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:26.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:27.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:28.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:28.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:28.607 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:13:28.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:13:28.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:28.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:29.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:30.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:30.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:30.330 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:13:30.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:13:30.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:30.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:31.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:32.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:32.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:32.682 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:13:32.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:13:32.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:32.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:33.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:34.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:34.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:34.544 INFO:tasks.workunit.client.0.smithi130.stdout:19 2024-08-19T11:13:34.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T11:13:34.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:34.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:35.012 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2437565342,v1:127.0.0.1:6801/2437565342] [v2:127.0.0.1:6802/2437565342,v1:127.0.0.1:6803/2437565342] exists,up 929108ec-bc3e-43f0-9730-8a494c85d7a3 2024-08-19T11:13:35.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:13:35.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:13:35.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:13:35.013 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-08-19T11:13:35.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2024-08-19T11:13:35.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-08-19T11:13:35.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:13:35.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:13:35.015 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:13:35.015 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:35.015 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:35.015 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:13:35.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-08-19T11:13:35.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_max_backfills 2024-08-19T11:13:35.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2024-08-19T11:13:35.129 INFO:tasks.workunit.client.0.smithi130.stdout:osd_max_backfills: 1 2024-08-19T11:13:35.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2024-08-19T11:13:35.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-08-19T11:13:35.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-08-19T11:13:37.727 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-08-19T11:13:37.727 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2024-08-19T11:13:37.728 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-08-19T11:13:37.728 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:13:37.728 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:13:37.729 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:13:37.729 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:37.729 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:37.729 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:13:37.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-08-19T11:13:37.730 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_max_backfills 2024-08-19T11:13:37.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2024-08-19T11:13:37.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-08-19T11:13:37.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-08-19T11:13:37.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2024-08-19T11:13:37.854 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-08-19T11:13:37.854 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:13:37.854 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:13:37.855 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:13:37.855 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:37.855 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:37.855 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:13:37.855 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-08-19T11:13:37.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok dump_recovery_reservations 2024-08-19T11:13:37.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-08-19T11:13:37.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-08-19T11:13:37.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2024-08-19T11:13:37.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-08-19T11:13:37.965 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-08-19T11:13:37.965 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:13:37.965 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:13:37.965 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:13:37.965 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:37.966 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:37.966 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:13:37.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-08-19T11:13:37.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok dump_recovery_reservations 2024-08-19T11:13:38.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-08-19T11:13:38.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-08-19T11:13:38.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-08-19T11:13:38.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-08-19T11:13:39.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-08-19T11:13:41.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-08-19T11:13:41.051 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2024-08-19T11:13:41.051 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-08-19T11:13:41.051 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:13:41.051 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:13:41.052 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:13:41.052 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:41.052 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:41.052 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:13:41.053 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-08-19T11:13:41.053 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_max_backfills 2024-08-19T11:13:41.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-08-19T11:13:41.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-08-19T11:13:41.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-08-19T11:13:41.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2024-08-19T11:13:41.173 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-08-19T11:13:41.173 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:13:41.173 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:13:41.174 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:13:41.174 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:41.174 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:41.174 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:13:41.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-08-19T11:13:41.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok dump_recovery_reservations 2024-08-19T11:13:41.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-08-19T11:13:41.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-08-19T11:13:41.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-08-19T11:13:41.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2024-08-19T11:13:41.293 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-08-19T11:13:41.294 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:13:41.294 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:13:41.294 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:13:41.295 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:41.295 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:41.295 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:13:41.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-08-19T11:13:41.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok dump_recovery_reservations 2024-08-19T11:13:41.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-08-19T11:13:41.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-08-19T11:13:41.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2024-08-19T11:13:41.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:13:41.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:13:41.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:13:41.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:13:41.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:13:41.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:13:41.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2024-08-19T11:13:42.196 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already down. 2024-08-19T11:13:42.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2024-08-19T11:13:42.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-08-19T11:13:42.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:13:42.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:13:42.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:13:42.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:42.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:13:42.214 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:13:42.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:42.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-08-19T11:13:42.663 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6800/2437565342,v1:127.0.0.1:6801/2437565342] [v2:127.0.0.1:6802/2437565342,v1:127.0.0.1:6803/2437565342] exists 929108ec-bc3e-43f0-9730-8a494c85d7a3 2024-08-19T11:13:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:13:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:13:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:13:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2024-08-19T11:13:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/mclock-config 2024-08-19T11:13:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T11:13:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T11:13:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T11:13:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/mclock-config/0 2024-08-19T11:13:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-08-19T11:13:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:13:42.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:13:42.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:13:42.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-08-19T11:13:42.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-08-19T11:13:42.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T11:13:42.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T11:13:42.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-08-19T11:13:42.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T11:13:42.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:13:42.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:13:42.668 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:13:42.668 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:42.668 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:42.669 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:13:42.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:13:42.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:13:42.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-08-19T11:13:42.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-08-19T11:13:42.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:13:42.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:13:42.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:13:42.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:13:42.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T11:13:42.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2024-08-19T11:13:42.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/mclock-config/0 2024-08-19T11:13:42.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T11:13:42.670 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:13:42.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2024-08-19T11:13:42.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/mclock-config/0/whoami 2024-08-19T11:13:42.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T11:13:42.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T11:13:42.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T11:13:42.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T11:13:42.692 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:42.688+0000 7f42e2f4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:13:42.696 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:42.692+0000 7f42e2f4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:13:42.697 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:42.696+0000 7f42e2f4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:13:43.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T11:13:43.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:13:43.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:13:43.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:13:43.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:13:43.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:43.114 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:13:43.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:13:43.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:43.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:43.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:43.948 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:43.944+0000 7f42e2f4e7c0 -1 Falling back to public interface 2024-08-19T11:13:44.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:44.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:44.608 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:13:44.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:13:44.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:44.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:45.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:46.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:46.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:46.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:13:46.061 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:13:46.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:46.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:46.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:47.081 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:47.075+0000 7f42e2f4e7c0 -1 osd.0 6 log_to_monitors true 2024-08-19T11:13:47.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:47.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:47.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:13:47.518 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:13:47.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:47.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:48.018 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6800/2473385278,v1:127.0.0.1:6801/2473385278] [v2:127.0.0.1:6802/2473385278,v1:127.0.0.1:6803/2473385278] exists,up 929108ec-bc3e-43f0-9730-8a494c85d7a3 2024-08-19T11:13:48.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:13:48.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:13:48.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:13:48.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2024-08-19T11:13:48.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-08-19T11:13:48.020 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-08-19T11:13:48.020 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:13:48.020 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:13:48.020 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:13:48.020 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:48.020 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:48.020 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:13:48.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-08-19T11:13:48.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_max_backfills 2024-08-19T11:13:48.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-08-19T11:13:48.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-08-19T11:13:48.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-08-19T11:13:48.140 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2024-08-19T11:13:48.140 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-08-19T11:13:48.140 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:13:48.140 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:13:48.140 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:13:48.140 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:48.140 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:48.141 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:13:48.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-08-19T11:13:48.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok dump_recovery_reservations 2024-08-19T11:13:48.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-08-19T11:13:48.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-08-19T11:13:48.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-08-19T11:13:48.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2024-08-19T11:13:48.250 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-08-19T11:13:48.250 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:13:48.250 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:13:48.251 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:13:48.251 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:48.251 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:48.251 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:13:48.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-08-19T11:13:48.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok dump_recovery_reservations 2024-08-19T11:13:48.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-08-19T11:13:48.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-08-19T11:13:48.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2024-08-19T11:13:48.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:13:48.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:13:48.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:13:48.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:13:48.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:13:48.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:13:48.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:13:48.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:13:48.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:13:48.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:13:48.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:13:48.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:13:48.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:13:48.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:13:48.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:13:48.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:13:48.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:13:48.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:13:48.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:13:48.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:13:48.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:13:48.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:13:48.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:13:48.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:48.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:48.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:13:48.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:13:48.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:13:48.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-08-19T11:13:48.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:13:48.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:13:48.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:13:48.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:13:48.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:13:48.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:13:48.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:13:48.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:13:48.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:13:48.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:13:48.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:13:48.694 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:13:48.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:13:48.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:13:48.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:13:48.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:13:48.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:13:48.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:13:48.697 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:13:48.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:13:48.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:13:48.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:13:48.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:13:48.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:48.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:48.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:13:48.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:13:48.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:13:48.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-08-19T11:13:48.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-08-19T11:13:48.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-08-19T11:13:48.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-08-19T11:13:48.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:13:48.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:13:48.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:13:48.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:13:48.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:13:48.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:13:48.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:13:48.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:13:48.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:13:48.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:13:48.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:13:48.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:13:48.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:13:48.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:13:48.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:13:48.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:13:48.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:13:48.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:13:48.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:13:48.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:13:48.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:13:48.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:13:48.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:13:48.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:48.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:48.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:13:48.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:13:48.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:13:48.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-08-19T11:13:48.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:13:48.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:48.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:48.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21434 2024-08-19T11:13:48.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:13:48.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:13:48.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:13:48.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-08-19T11:13:48.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2024-08-19T11:13:48.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2024-08-19T11:13:48.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2024-08-19T11:13:48.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-08-19T11:13:48.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:13:48.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:13:48.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:13:48.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-08-19T11:13:48.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-08-19T11:13:49.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:13:49.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:13:49.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:13:49.177 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:13:49.177 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:49.177 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:49.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:13:49.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:13:49.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:13:49.373 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:13:49.374 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:13:49.374 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:13:49.374 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:13:49.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:13:49.375 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:13:49.375 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:13:49.375 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:13:49.375 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:13:49.375 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:49.375 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:49.375 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-mon.a.asok 2024-08-19T11:13:49.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:13:49.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21434/ceph-mon.a.asok config get fsid 2024-08-19T11:13:49.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:13:49.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:13:49.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:13:49.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:13:49.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:13:49.493 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:13:49.494 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:13:49.494 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:13:49.495 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:13:49.495 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:49.495 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:49.495 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-mon.a.asok 2024-08-19T11:13:49.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:13:49.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21434/ceph-mon.a.asok config get mon_host 2024-08-19T11:13:49.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2024-08-19T11:13:49.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-08-19T11:13:49.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:13:49.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:13:49.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:13:49.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-08-19T11:13:49.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:13:50.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:13:50.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:13:50.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:13:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:13:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:50.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:13:50.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:13:50.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:13:50.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-08-19T11:13:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-08-19T11:13:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:13:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:13:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:13:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-08-19T11:13:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-08-19T11:13:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:13:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:13:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:13:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-08-19T11:13:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-08-19T11:13:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:13:50.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-08-19T11:13:50.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-08-19T11:13:50.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:13:50.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:13:50.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:13:50.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:13:50.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:13:50.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-08-19T11:13:50.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-08-19T11:13:50.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:13:50.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a334571d-7e28-43cf-8b30-e7b3ca8d3c0c 2024-08-19T11:13:50.543 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 a334571d-7e28-43cf-8b30-e7b3ca8d3c0c 2024-08-19T11:13:50.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a334571d-7e28-43cf-8b30-e7b3ca8d3c0c' 2024-08-19T11:13:50.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:13:50.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDuKMNmEgBBIRAAhyEJ9LmksYOIx2Rqir6V9g== 2024-08-19T11:13:50.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDuKMNmEgBBIRAAhyEJ9LmksYOIx2Rqir6V9g=="}' 2024-08-19T11:13:50.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a334571d-7e28-43cf-8b30-e7b3ca8d3c0c -i td/mclock-config/0/new.json 2024-08-19T11:13:50.849 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:13:50.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-08-19T11:13:50.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDuKMNmEgBBIRAAhyEJ9LmksYOIx2Rqir6V9g== --osd-uuid a334571d-7e28-43cf-8b30-e7b3ca8d3c0c 2024-08-19T11:13:50.900 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:50.895+0000 7f4ed87467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:13:50.918 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:50.915+0000 7f4ed87467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:13:50.919 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:50.915+0000 7f4ed87467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:13:50.920 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:50.915+0000 7f4ed87467c0 -1 bdev(0x5599d6b43c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:13:50.920 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:50.915+0000 7f4ed87467c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-08-19T11:13:56.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-08-19T11:13:56.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:13:56.864 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:13:56.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:13:56.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:13:57.445 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:13:57.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:13:57.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-08-19T11:13:57.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:13:57.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:13:57.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:13:57.470 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:57.467+0000 7f2be5f077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:13:57.473 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:57.467+0000 7f2be5f077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:13:57.474 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:57.471+0000 7f2be5f077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:13:57.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:13:57.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:13:57.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:13:57.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:13:57.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:13:57.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:57.922 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:13:57.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:13:57.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:57.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:58.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:13:58.780 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:13:58.775+0000 7f2be5f077c0 -1 Falling back to public interface 2024-08-19T11:13:59.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:13:59.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:13:59.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:13:59.356 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:13:59.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:13:59.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:13:59.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:00.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:00.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:00.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:14:00.771 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:14:00.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:00.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:01.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:01.789 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:14:01.783+0000 7f2be5f077c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:14:02.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:02.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:02.181 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:14:02.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:14:02.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:02.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:02.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:03.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:03.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:03.745 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:14:03.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:14:03.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:03.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:04.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:05.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:05.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:05.162 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:14:05.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:14:05.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:05.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:05.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:06.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:06.630 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:14:06.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:06.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:14:06.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:06.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:07.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:08.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:08.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:08.074 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:14:08.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:14:08.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:08.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:08.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:09.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:09.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:09.689 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:14:09.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:14:09.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:09.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:10.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:11.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:11.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:11.130 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:14:11.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:14:11.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:11.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:11.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:12.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:12.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:12.648 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:14:12.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:14:12.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:12.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:13.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:14.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:14.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:14.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:14:14.265 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:14:14.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:14.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:15.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:16.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:16.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:16.293 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:14:16.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:14:16.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:16.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:17.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:18.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:18.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:18.438 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:14:18.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:14:18.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:18.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:19.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:20.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:20.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:20.413 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:14:20.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:14:20.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:20.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:21.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:22.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:22.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:22.482 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:14:22.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:14:22.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:22.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:23.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:24.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:24.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:24.236 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:14:24.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:14:24.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:24.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:24.710 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1500198928,v1:127.0.0.1:6803/1500198928] [v2:127.0.0.1:6804/1500198928,v1:127.0.0.1:6805/1500198928] exists,up a334571d-7e28-43cf-8b30-e7b3ca8d3c0c 2024-08-19T11:14:24.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:14:24.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:14:24.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:14:24.711 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2024-08-19T11:14:25.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2024-08-19T11:14:25.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2024-08-19T11:14:25.161 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-08-19T11:14:25.162 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-08-19T11:14:25.162 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:14:25.162 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:14:25.162 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:14:25.162 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:25.163 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:25.163 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:14:25.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-08-19T11:14:25.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_profile 2024-08-19T11:14:25.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2024-08-19T11:14:25.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2024-08-19T11:14:25.284 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2024-08-19T11:14:25.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2024-08-19T11:14:25.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2024-08-19T11:14:25.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2024-08-19T11:14:25.427 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:14:25.428 INFO:tasks.workunit.client.0.smithi130.stdout: "success": "" 2024-08-19T11:14:25.428 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:14:25.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-08-19T11:14:25.443 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-08-19T11:14:25.443 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:14:25.443 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:14:25.444 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:14:25.444 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:25.444 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:25.444 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:14:25.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-08-19T11:14:25.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_profile 2024-08-19T11:14:25.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2024-08-19T11:14:25.551 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2024-08-19T11:14:25.551 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2024-08-19T11:14:25.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2024-08-19T11:14:25.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2024-08-19T11:14:25.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2024-08-19T11:14:25.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-08-19T11:14:25.553 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-08-19T11:14:25.553 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:14:25.553 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:14:25.554 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:14:25.554 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:25.554 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:25.554 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:14:25.554 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-08-19T11:14:25.554 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-08-19T11:14:25.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2024-08-19T11:14:25.667 INFO:tasks.workunit.client.0.smithi130.stdout:client_res = 0.300000 2024-08-19T11:14:25.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2024-08-19T11:14:25.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2024-08-19T11:14:25.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2024-08-19T11:14:25.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2024-08-19T11:14:25.670 INFO:tasks.workunit.client.0.smithi130.stdout:client_res_new = .400000 2024-08-19T11:14:25.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2024-08-19T11:14:25.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-08-19T11:14:26.161 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-08-19T11:14:26.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2024-08-19T11:14:26.631 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-08-19T11:14:26.631 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2024-08-19T11:14:26.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2024-08-19T11:14:26.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-08-19T11:14:26.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2024-08-19T11:14:26.636 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-08-19T11:14:26.636 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:14:26.636 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:14:26.636 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:14:26.636 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:26.636 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:26.636 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:14:26.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-08-19T11:14:26.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-08-19T11:14:26.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2024-08-19T11:14:26.743 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-08-19T11:14:26.743 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2024-08-19T11:14:26.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2024-08-19T11:14:26.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2024-08-19T11:14:26.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:14:26.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:14:26.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:14:26.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:14:26.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:14:26.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:14:26.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:14:26.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:14:26.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:14:26.855 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:14:26.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:14:26.855 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:14:26.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:14:26.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:14:26.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:14:26.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:14:26.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:14:26.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:14:26.858 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:14:26.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:14:26.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:14:26.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:14:26.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:14:26.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:26.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:26.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:14:26.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:14:26.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:14:26.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-08-19T11:14:26.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:14:26.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:14:26.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:14:26.867 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:14:26.867 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:14:26.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:14:26.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:14:26.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:14:26.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:14:26.868 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:14:26.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:14:26.869 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:14:26.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:14:26.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:14:26.871 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:14:26.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:14:26.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:14:26.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:14:26.872 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:14:26.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:14:26.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:14:26.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:14:26.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:14:26.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:26.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:26.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:14:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:14:26.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:14:26.878 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:14:26.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:14:26.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:14:26.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:14:26.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:14:26.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:14:26.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:14:26.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:14:26.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:14:26.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:14:26.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:14:26.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:14:26.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:14:26.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:14:26.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:26.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:26.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:14:26.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:14:26.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:14:26.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-08-19T11:14:26.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:14:26.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:26.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:26.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21434 2024-08-19T11:14:26.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:14:26.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:14:26.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:14:26.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-08-19T11:14:26.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2024-08-19T11:14:26.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2024-08-19T11:14:26.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2024-08-19T11:14:26.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-08-19T11:14:26.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-08-19T11:14:26.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:14:26.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:14:26.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:14:26.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:14:26.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:14:26.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:14:26.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:14:26.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:14:26.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:14:26.890 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:14:26.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:14:26.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:14:26.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:14:26.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:14:26.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:14:26.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:14:26.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:14:26.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:14:26.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:14:26.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:14:26.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:14:26.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:14:26.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:14:26.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:26.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:26.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:14:26.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:14:26.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:14:26.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-08-19T11:14:26.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:14:26.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:26.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:26.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21434 2024-08-19T11:14:26.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:14:26.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:14:26.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:14:26.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-08-19T11:14:26.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2024-08-19T11:14:26.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-08-19T11:14:26.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:14:26.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:14:26.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:14:26.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-08-19T11:14:26.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-08-19T11:14:27.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:14:27.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:14:27.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:14:27.389 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:14:27.390 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:27.390 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:27.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:14:27.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:14:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:14:27.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:14:27.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:14:27.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:14:27.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:14:27.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:14:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:14:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:14:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:14:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:14:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:27.588 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-mon.a.asok 2024-08-19T11:14:27.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:14:27.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21434/ceph-mon.a.asok config get fsid 2024-08-19T11:14:27.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:14:27.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:14:27.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:14:27.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:14:27.706 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:14:27.707 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:14:27.707 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:14:27.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:14:27.707 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:14:27.707 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:27.707 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:27.707 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-mon.a.asok 2024-08-19T11:14:27.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:14:27.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21434/ceph-mon.a.asok config get mon_host 2024-08-19T11:14:27.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2024-08-19T11:14:27.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-08-19T11:14:27.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:14:27.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:14:27.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:14:27.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-08-19T11:14:27.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:14:28.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:14:28.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:14:28.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:14:28.745 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:14:28.745 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:28.745 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:28.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:14:28.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:14:28.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:14:28.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-08-19T11:14:28.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-08-19T11:14:28.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:14:28.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:14:28.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:14:28.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-08-19T11:14:28.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-08-19T11:14:28.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:14:28.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:14:28.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:14:28.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-08-19T11:14:28.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-08-19T11:14:28.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:14:28.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-08-19T11:14:28.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-08-19T11:14:28.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:14:28.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:14:28.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:14:28.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:14:28.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:14:28.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-08-19T11:14:28.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-08-19T11:14:28.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:14:28.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a0c25903-20fd-4ba5-b0e8-4c3bc1212a3f 2024-08-19T11:14:28.784 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 a0c25903-20fd-4ba5-b0e8-4c3bc1212a3f 2024-08-19T11:14:28.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a0c25903-20fd-4ba5-b0e8-4c3bc1212a3f' 2024-08-19T11:14:28.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:14:28.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUKcNms16YLxAANJEwdlbeY+Z9I0srbUfe0Q== 2024-08-19T11:14:28.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUKcNms16YLxAANJEwdlbeY+Z9I0srbUfe0Q=="}' 2024-08-19T11:14:28.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a0c25903-20fd-4ba5-b0e8-4c3bc1212a3f -i td/mclock-config/0/new.json 2024-08-19T11:14:29.086 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:14:29.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-08-19T11:14:29.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAUKcNms16YLxAANJEwdlbeY+Z9I0srbUfe0Q== --osd-uuid a0c25903-20fd-4ba5-b0e8-4c3bc1212a3f 2024-08-19T11:14:29.130 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:14:29.130+0000 7fc7303f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:14:29.155 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:14:29.154+0000 7fc7303f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:14:29.156 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:14:29.154+0000 7fc7303f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:14:29.157 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:14:29.154+0000 7fc7303f57c0 -1 bdev(0x557dc8ecdc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:14:29.157 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:14:29.154+0000 7fc7303f57c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-08-19T11:14:35.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-08-19T11:14:35.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:14:35.058 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:14:35.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:14:35.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:14:35.659 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:14:35.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:14:35.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-08-19T11:14:35.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:14:35.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:14:35.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:14:35.690 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:14:35.690+0000 7f201afa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:14:35.692 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:14:35.690+0000 7f201afa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:14:35.693 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:14:35.690+0000 7f201afa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:14:36.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:14:36.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:14:36.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:14:36.112 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:14:36.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:14:36.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:14:36.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:36.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:14:36.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:36.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:36.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:36.904 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:14:36.902+0000 7f201afa37c0 -1 Falling back to public interface 2024-08-19T11:14:37.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:37.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:37.609 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:14:37.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:14:37.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:37.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:38.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:39.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:39.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:39.083 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:14:39.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:14:39.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:39.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:39.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:39.594 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:14:39.594+0000 7f201afa37c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:14:40.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:40.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:40.524 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:14:40.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:14:40.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:40.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:40.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:41.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:41.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:41.999 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:14:41.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:14:41.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:42.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:42.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:43.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:43.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:43.546 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:14:43.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:14:43.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:43.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:43.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:44.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:44.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:44.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:14:44.993 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:14:44.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:44.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:45.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:46.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:46.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:46.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:14:46.525 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:14:46.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:46.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:46.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:47.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:47.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:47.964 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:14:47.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:14:47.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:47.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:48.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:49.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:49.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:49.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:14:49.496 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:14:49.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:49.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:49.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:50.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:50.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:50.952 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:14:50.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:14:50.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:50.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:51.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:52.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:52.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:52.728 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:14:52.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:14:52.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:52.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:53.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:54.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:54.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:54.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:14:54.490 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:14:54.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:54.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:55.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:56.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:56.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:56.197 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:14:56.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:14:56.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:56.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:57.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:58.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:58.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:58.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:14:58.144 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:14:58.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:58.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:14:58.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:14:59.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:14:59.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:14:59.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:14:59.909 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:14:59.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:14:59.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:01.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:02.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:02.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:02.003 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:15:02.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:15:02.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:02.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:02.465 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3785761100,v1:127.0.0.1:6803/3785761100] [v2:127.0.0.1:6804/3785761100,v1:127.0.0.1:6805/3785761100] exists,up a0c25903-20fd-4ba5-b0e8-4c3bc1212a3f 2024-08-19T11:15:02.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:15:02.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:15:02.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:15:02.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2024-08-19T11:15:02.466 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2024-08-19T11:15:02.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2024-08-19T11:15:02.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2024-08-19T11:15:02.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-08-19T11:15:02.896 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-08-19T11:15:02.896 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:15:02.896 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:15:02.897 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:02.897 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:02.897 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:02.897 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:15:02.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-08-19T11:15:02.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_profile 2024-08-19T11:15:03.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2024-08-19T11:15:03.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2024-08-19T11:15:03.020 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2024-08-19T11:15:03.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2024-08-19T11:15:03.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-08-19T11:15:03.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2024-08-19T11:15:03.160 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:15:03.160 INFO:tasks.workunit.client.0.smithi130.stdout: "success": "" 2024-08-19T11:15:03.160 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:15:03.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-08-19T11:15:03.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-08-19T11:15:03.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:15:03.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:15:03.176 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:03.176 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:03.176 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:03.176 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:15:03.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-08-19T11:15:03.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_profile 2024-08-19T11:15:03.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2024-08-19T11:15:03.282 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2024-08-19T11:15:03.282 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2024-08-19T11:15:03.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2024-08-19T11:15:03.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2024-08-19T11:15:03.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2024-08-19T11:15:03.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2024-08-19T11:15:03.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-08-19T11:15:03.284 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-08-19T11:15:03.284 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:15:03.285 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:15:03.285 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:03.285 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:03.285 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:03.285 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:15:03.286 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-08-19T11:15:03.286 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-08-19T11:15:03.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2024-08-19T11:15:03.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2024-08-19T11:15:03.400 INFO:tasks.workunit.client.0.smithi130.stdout:Original client_res for osd.0 = 0.300000 2024-08-19T11:15:03.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2024-08-19T11:15:03.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2024-08-19T11:15:03.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2024-08-19T11:15:03.402 INFO:tasks.workunit.client.0.smithi130.stdout:client_res_new = .400000 2024-08-19T11:15:03.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2024-08-19T11:15:03.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2024-08-19T11:15:03.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-08-19T11:15:04.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2024-08-19T11:15:04.362 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-08-19T11:15:04.362 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2024-08-19T11:15:04.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2024-08-19T11:15:04.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-08-19T11:15:04.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2024-08-19T11:15:04.365 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-08-19T11:15:04.365 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:15:04.365 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:15:04.366 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:04.366 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:04.366 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:04.366 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:15:04.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-08-19T11:15:04.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-08-19T11:15:04.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2024-08-19T11:15:04.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-08-19T11:15:04.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2024-08-19T11:15:04.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2024-08-19T11:15:04.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2024-08-19T11:15:04.615 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:15:04.615 INFO:tasks.workunit.client.0.smithi130.stdout: "success": "" 2024-08-19T11:15:04.615 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:15:04.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-08-19T11:15:04.630 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-08-19T11:15:04.630 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:15:04.630 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:15:04.630 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:04.630 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:04.630 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:04.630 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:15:04.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-08-19T11:15:04.631 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_profile 2024-08-19T11:15:04.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2024-08-19T11:15:04.744 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2024-08-19T11:15:04.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-08-19T11:15:04.744 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-08-19T11:15:05.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2024-08-19T11:15:05.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2024-08-19T11:15:05.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-08-19T11:15:05.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2024-08-19T11:15:05.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-08-19T11:15:05.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2024-08-19T11:15:05.152 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-08-19T11:15:05.152 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:15:05.152 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:15:05.152 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:05.153 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:05.153 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:05.153 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:15:05.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-08-19T11:15:05.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-08-19T11:15:05.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2024-08-19T11:15:05.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-08-19T11:15:05.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2024-08-19T11:15:05.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2024-08-19T11:15:05.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2024-08-19T11:15:05.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2024-08-19T11:15:10.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-08-19T11:15:11.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2024-08-19T11:15:11.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2024-08-19T11:15:11.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2024-08-19T11:15:11.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2024-08-19T11:15:11.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-08-19T11:15:11.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2024-08-19T11:15:11.228 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-08-19T11:15:11.228 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:15:11.228 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:15:11.229 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:11.229 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:11.229 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:11.230 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:15:11.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-08-19T11:15:11.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-08-19T11:15:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2024-08-19T11:15:11.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2024-08-19T11:15:11.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2024-08-19T11:15:11.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2024-08-19T11:15:11.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2024-08-19T11:15:11.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:15:11.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:15:11.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:15:11.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:15:11.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:15:11.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:15:11.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:15:11.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:15:11.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:15:11.665 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:15:11.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:15:11.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:15:11.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:15:11.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:15:11.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:15:11.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:11.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:15:11.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:15:11.670 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:11.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:15:11.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:15:11.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:15:11.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:15:11.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:11.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:11.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:15:11.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:15:11.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:15:11.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-08-19T11:15:11.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:15:11.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:15:11.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:15:11.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:15:11.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:15:11.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:15:11.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:15:11.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:15:11.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:15:11.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:15:11.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:15:11.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:15:11.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:15:11.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:15:11.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:15:11.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:11.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:15:11.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:15:11.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:11.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:15:11.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:15:11.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:15:11.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:15:11.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:11.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:11.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:15:11.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:15:11.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:15:11.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-08-19T11:15:11.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-08-19T11:15:11.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-08-19T11:15:11.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-08-19T11:15:11.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:15:11.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:15:11.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:15:11.699 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:15:11.699 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:15:11.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:15:11.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:15:11.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:15:11.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:15:11.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:15:11.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:15:11.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:15:11.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:15:11.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:15:11.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:15:11.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:11.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:15:11.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:15:11.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:11.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:15:11.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:15:11.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:15:11.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:15:11.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:11.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:11.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:15:11.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:15:11.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:15:11.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-08-19T11:15:11.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:15:11.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:11.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:11.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21434 2024-08-19T11:15:11.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:15:11.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:15:11.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:15:11.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:15:11.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:15:11.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:15:11.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:15:11.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:15:11.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:15:11.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:15:11.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:15:11.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:11.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:15:11.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:15:11.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:11.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:15:11.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:15:11.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:15:11.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:15:11.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:11.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:11.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:15:11.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:15:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:15:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-08-19T11:15:11.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:15:11.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:11.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:11.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21434 2024-08-19T11:15:11.731 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:15:11.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:15:11.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:15:11.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-08-19T11:15:11.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2024-08-19T11:15:11.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-08-19T11:15:11.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:15:11.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:15:11.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:15:11.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-08-19T11:15:11.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-08-19T11:15:12.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:15:12.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:15:12.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:15:12.192 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:15:12.192 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:12.192 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:12.192 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:15:12.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:15:12.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:15:12.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:15:12.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:15:12.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:15:12.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:15:12.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:15:12.371 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:15:12.371 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:15:12.371 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:15:12.371 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:12.372 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:12.372 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:12.372 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-mon.a.asok 2024-08-19T11:15:12.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:15:12.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21434/ceph-mon.a.asok config get fsid 2024-08-19T11:15:12.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:15:12.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:15:12.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:15:12.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:15:12.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:15:12.487 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:15:12.488 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:15:12.488 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:15:12.488 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:12.488 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:12.488 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:12.488 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-mon.a.asok 2024-08-19T11:15:12.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:15:12.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21434/ceph-mon.a.asok config get mon_host 2024-08-19T11:15:12.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2024-08-19T11:15:12.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-08-19T11:15:12.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:15:12.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:15:12.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:15:12.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-08-19T11:15:12.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:15:13.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:15:13.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:15:13.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:15:13.498 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:15:13.498 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:13.498 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:13.498 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:15:13.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:15:13.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:15:13.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-08-19T11:15:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-08-19T11:15:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:15:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:15:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:15:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-08-19T11:15:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-08-19T11:15:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:15:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:15:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:15:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-08-19T11:15:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-08-19T11:15:13.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:15:13.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:15:13.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-08-19T11:15:13.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:15:13.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:15:13.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:15:13.536 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:15:13.536 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:13.536 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:13.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:15:13.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:15:13.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:15:13.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:15:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:15:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-08-19T11:15:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-08-19T11:15:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:15:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:15:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:15:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:15:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:15:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-08-19T11:15:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-08-19T11:15:13.538 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:15:13.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1814d3b6-a833-4032-bd75-3fd9bf092048 2024-08-19T11:15:13.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1814d3b6-a833-4032-bd75-3fd9bf092048' 2024-08-19T11:15:13.538 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 1814d3b6-a833-4032-bd75-3fd9bf092048 2024-08-19T11:15:13.538 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:15:13.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBKcNmO5vvIBAA2iVYtYl9qjDQp8fiyP/EdA== 2024-08-19T11:15:13.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBKcNmO5vvIBAA2iVYtYl9qjDQp8fiyP/EdA=="}' 2024-08-19T11:15:13.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1814d3b6-a833-4032-bd75-3fd9bf092048 -i td/mclock-config/0/new.json 2024-08-19T11:15:13.838 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:15:13.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-08-19T11:15:13.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBBKcNmO5vvIBAA2iVYtYl9qjDQp8fiyP/EdA== --osd-uuid 1814d3b6-a833-4032-bd75-3fd9bf092048 2024-08-19T11:15:13.894 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:13.889+0000 7f113f7357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:15:13.898 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:13.897+0000 7f113f7357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:15:13.900 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:13.897+0000 7f113f7357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:15:13.900 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:13.897+0000 7f113f7357c0 -1 bdev(0x562f9c5a3c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:15:13.900 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:13.897+0000 7f113f7357c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-08-19T11:15:19.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-08-19T11:15:19.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:15:19.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:15:19.835 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:15:19.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:15:20.421 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:15:20.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:15:20.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:15:20.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:15:20.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-08-19T11:15:20.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:15:20.452 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:20.449+0000 7f27d402d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:15:20.454 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:20.449+0000 7f27d402d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:15:20.455 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:20.453+0000 7f27d402d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:15:20.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:15:20.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:15:20.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:15:20.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:15:20.873 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:15:20.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:15:20.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:20.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:15:20.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:20.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:21.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:21.669 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:21.665+0000 7f27d402d7c0 -1 Falling back to public interface 2024-08-19T11:15:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:22.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:22.295 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:15:22.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:15:22.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:22.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:22.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:23.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:23.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:23.714 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:15:23.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:15:23.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:23.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:24.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:24.443 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:24.440+0000 7f27d402d7c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:15:25.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:25.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:25.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:15:25.142 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:15:25.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:25.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:25.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:26.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:26.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:26.618 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:15:26.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:15:26.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:26.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:27.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:28.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:28.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:28.042 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:15:28.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:15:28.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:28.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:28.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:29.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:29.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:29.575 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:15:29.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:15:29.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:29.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:30.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:31.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:31.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:31.023 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:15:31.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:15:31.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:31.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:31.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:32.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:32.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:32.599 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:15:32.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:15:32.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:32.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:33.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:34.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:34.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:34.050 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:15:34.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:15:34.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:34.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:34.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:35.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:35.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:35.567 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:15:35.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:15:35.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:35.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:36.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:37.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:37.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:37.050 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:15:37.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:15:37.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:37.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:37.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:38.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:38.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:38.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:15:38.596 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:15:38.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:38.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:39.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:40.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:40.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:40.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:15:40.556 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:15:40.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:40.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:41.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:42.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:42.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:42.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:15:42.939 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:15:42.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:42.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:43.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:44.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:44.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:44.902 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:15:44.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:15:44.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:44.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:45.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:15:46.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:15:46.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:15:46.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:15:46.956 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:15:46.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:15:46.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:15:47.427 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2939387674,v1:127.0.0.1:6803/2939387674] [v2:127.0.0.1:6804/2939387674,v1:127.0.0.1:6805/2939387674] exists,up 1814d3b6-a833-4032-bd75-3fd9bf092048 2024-08-19T11:15:47.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:15:47.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:15:47.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:15:47.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2024-08-19T11:15:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2024-08-19T11:15:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2024-08-19T11:15:47.988 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-08-19T11:15:47.988 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:15:47.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2024-08-19T11:15:47.988 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:15:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:15:47.989 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-08-19T11:15:47.989 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_profile 2024-08-19T11:15:48.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2024-08-19T11:15:48.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2024-08-19T11:15:48.100 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2024-08-19T11:15:48.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2024-08-19T11:15:48.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2024-08-19T11:15:48.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-08-19T11:15:48.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2024-08-19T11:15:48.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2024-08-19T11:15:48.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2024-08-19T11:15:48.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-08-19T11:15:48.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-08-19T11:15:48.102 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-08-19T11:15:48.102 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:15:48.102 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:15:48.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-08-19T11:15:48.103 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:48.103 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:48.103 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:48.103 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:15:48.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-08-19T11:15:48.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-08-19T11:15:48.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2024-08-19T11:15:48.208 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2024-08-19T11:15:48.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-08-19T11:15:48.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2024-08-19T11:15:48.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2024-08-19T11:15:48.782 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-08-19T11:15:48.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-08-19T11:15:48.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-08-19T11:15:48.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-08-19T11:15:48.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-08-19T11:15:50.820 INFO:tasks.workunit.client.0.smithi130.stdout:Check configuration values - Attempt#: 0 2024-08-19T11:15:50.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-08-19T11:15:50.821 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-08-19T11:15:51.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-08-19T11:15:51.273 INFO:tasks.workunit.client.0.smithi130.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-08-19T11:15:51.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-08-19T11:15:51.273 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2024-08-19T11:15:51.273 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-08-19T11:15:51.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-08-19T11:15:51.276 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-08-19T11:15:51.276 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2024-08-19T11:15:51.276 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-08-19T11:15:51.276 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-08-19T11:15:51.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2024-08-19T11:15:51.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-08-19T11:15:51.728 INFO:tasks.workunit.client.0.smithi130.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-08-19T11:15:51.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2024-08-19T11:15:51.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-08-19T11:15:51.730 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2024-08-19T11:15:51.730 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-08-19T11:15:51.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-08-19T11:15:51.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-08-19T11:15:51.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-08-19T11:15:51.733 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-08-19T11:15:51.733 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:15:51.733 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:15:51.734 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:51.734 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:51.734 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:51.734 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:15:51.734 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-08-19T11:15:51.734 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-08-19T11:15:51.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2024-08-19T11:15:51.851 INFO:tasks.workunit.client.0.smithi130.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-08-19T11:15:51.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-08-19T11:15:51.852 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2024-08-19T11:15:51.852 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-08-19T11:15:51.855 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2024-08-19T11:15:51.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-08-19T11:15:51.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-08-19T11:15:51.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-08-19T11:15:51.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-08-19T11:15:51.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-08-19T11:15:51.858 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-08-19T11:15:51.858 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-08-19T11:15:51.858 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-08-19T11:15:51.858 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:15:51.858 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:15:51.859 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:51.859 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:51.859 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:51.860 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:15:51.860 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-08-19T11:15:51.860 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-08-19T11:15:51.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2024-08-19T11:15:51.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2024-08-19T11:15:51.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-08-19T11:15:51.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2024-08-19T11:15:51.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-08-19T11:15:52.588 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-08-19T11:15:52.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-08-19T11:15:52.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-08-19T11:15:52.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-08-19T11:15:52.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-08-19T11:15:54.593 INFO:tasks.workunit.client.0.smithi130.stdout:Check configuration values - Attempt#: 0 2024-08-19T11:15:54.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-08-19T11:15:54.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-08-19T11:15:55.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-08-19T11:15:55.030 INFO:tasks.workunit.client.0.smithi130.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-08-19T11:15:55.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-08-19T11:15:55.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-08-19T11:15:55.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-08-19T11:15:55.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-08-19T11:15:55.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-08-19T11:15:55.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-08-19T11:15:55.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-08-19T11:15:55.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-08-19T11:15:55.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2024-08-19T11:15:55.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-08-19T11:15:55.445 INFO:tasks.workunit.client.0.smithi130.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-08-19T11:15:55.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2024-08-19T11:15:55.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-08-19T11:15:55.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2024-08-19T11:15:55.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-08-19T11:15:55.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-08-19T11:15:55.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-08-19T11:15:55.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-08-19T11:15:55.449 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-08-19T11:15:55.449 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:15:55.449 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:15:55.450 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:55.450 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:55.450 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:55.450 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:15:55.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-08-19T11:15:55.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-08-19T11:15:55.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-08-19T11:15:55.569 INFO:tasks.workunit.client.0.smithi130.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-08-19T11:15:55.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-08-19T11:15:55.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-08-19T11:15:55.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-08-19T11:15:55.572 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-08-19T11:15:55.572 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-08-19T11:15:55.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-08-19T11:15:55.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-08-19T11:15:55.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-08-19T11:15:55.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2024-08-19T11:15:55.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:15:55.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:15:55.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:15:55.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:15:55.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:15:55.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:15:55.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:15:55.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:15:55.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:15:55.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:15:55.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:15:55.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:15:55.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:15:55.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:15:55.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:15:55.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:55.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:15:55.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:15:55.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:55.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:15:55.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:15:55.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:15:55.694 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:15:55.694 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:55.697 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:55.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:15:55.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:15:55.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:15:55.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-08-19T11:15:55.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:15:55.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:15:55.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:15:55.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:15:55.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:15:55.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:15:55.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:15:55.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:15:55.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:15:55.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:15:55.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:15:55.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:15:55.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:15:55.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:15:55.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:15:55.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:55.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:15:55.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:15:55.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:55.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:15:55.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:15:55.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:15:55.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:15:55.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:55.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:55.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:15:55.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:15:55.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:15:55.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-08-19T11:15:55.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-08-19T11:15:55.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-08-19T11:15:55.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-08-19T11:15:55.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:15:55.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:15:55.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:15:55.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:15:55.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:15:55.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:15:55.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:15:55.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:15:55.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:15:55.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:15:55.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:15:55.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:15:55.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:15:55.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:15:55.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:15:55.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:55.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:15:55.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:15:55.711 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:55.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:15:55.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:15:55.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:15:55.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:15:55.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:55.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:55.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:15:55.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:15:55.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:15:55.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-08-19T11:15:55.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:15:55.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:55.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:55.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21434 2024-08-19T11:15:55.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:15:55.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:15:55.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:15:55.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-08-19T11:15:55.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2024-08-19T11:15:55.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2024-08-19T11:15:55.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2024-08-19T11:15:55.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-08-19T11:15:55.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-08-19T11:15:55.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:15:55.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:15:55.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:15:55.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:15:55.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:15:55.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:15:55.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:15:55.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:15:55.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:15:55.720 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:15:55.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:15:55.720 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:15:55.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:15:55.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:15:55.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:15:55.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:55.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:15:55.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:15:55.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:15:55.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:15:55.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:15:55.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:15:55.724 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:15:55.725 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:55.725 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:55.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:15:55.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:15:55.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:15:55.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-08-19T11:15:55.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:15:55.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:55.727 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:55.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21434 2024-08-19T11:15:55.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:15:55.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:15:55.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:15:55.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-08-19T11:15:55.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2024-08-19T11:15:55.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-08-19T11:15:55.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:15:55.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:15:55.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:15:55.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-08-19T11:15:55.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-08-19T11:15:56.192 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:15:56.192 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:15:56.192 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:15:56.193 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:15:56.193 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:56.193 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:56.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:15:56.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:15:56.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:15:56.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:15:56.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:15:56.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:15:56.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:15:56.382 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:15:56.382 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:15:56.383 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:15:56.383 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:15:56.383 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:56.383 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:56.383 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:56.384 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-mon.a.asok 2024-08-19T11:15:56.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:15:56.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21434/ceph-mon.a.asok config get fsid 2024-08-19T11:15:56.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:15:56.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:15:56.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:15:56.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:15:56.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:15:56.496 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:15:56.496 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:15:56.496 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:15:56.497 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:15:56.497 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:56.497 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:56.498 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-mon.a.asok 2024-08-19T11:15:56.498 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:15:56.498 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21434/ceph-mon.a.asok config get mon_host 2024-08-19T11:15:56.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2024-08-19T11:15:56.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-08-19T11:15:56.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:15:56.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:15:56.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:15:56.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-08-19T11:15:56.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:15:57.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:15:57.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:15:57.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:15:57.529 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:15:57.529 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:57.529 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:57.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:15:57.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:15:57.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:15:57.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-08-19T11:15:57.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-08-19T11:15:57.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:15:57.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:15:57.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:15:57.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-08-19T11:15:57.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:15:57.566 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-08-19T11:15:57.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-08-19T11:15:57.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:15:57.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2abecb09-cce2-4be4-a9a9-1bbcc2dff847 2024-08-19T11:15:57.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2abecb09-cce2-4be4-a9a9-1bbcc2dff847' 2024-08-19T11:15:57.568 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 2abecb09-cce2-4be4-a9a9-1bbcc2dff847 2024-08-19T11:15:57.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:15:57.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtKcNmqEvCIhAAFnS4h2JaWvcYH5iYEcTbTA== 2024-08-19T11:15:57.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtKcNmqEvCIhAAFnS4h2JaWvcYH5iYEcTbTA=="}' 2024-08-19T11:15:57.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2abecb09-cce2-4be4-a9a9-1bbcc2dff847 -i td/mclock-config/0/new.json 2024-08-19T11:15:57.881 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:15:57.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-08-19T11:15:57.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBtKcNmqEvCIhAAFnS4h2JaWvcYH5iYEcTbTA== --osd-uuid 2abecb09-cce2-4be4-a9a9-1bbcc2dff847 2024-08-19T11:15:57.930 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:57.924+0000 7ff1b64bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:15:57.941 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:57.936+0000 7ff1b64bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:15:57.942 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:57.940+0000 7ff1b64bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:15:57.943 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:57.940+0000 7ff1b64bf7c0 -1 bdev(0x55baf4d41c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:15:57.943 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:15:57.940+0000 7ff1b64bf7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-08-19T11:16:03.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-08-19T11:16:03.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:16:03.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:16:03.979 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:16:03.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:16:04.568 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:16:04.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:16:04.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:16:04.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-08-19T11:16:04.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:16:04.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:16:04.600 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:04.596+0000 7f66c14ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:16:04.603 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:04.600+0000 7f66c14ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:16:04.604 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:04.600+0000 7f66c14ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:16:05.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:16:05.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:16:05.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:16:05.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:16:05.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:16:05.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:05.005 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:16:05.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:16:05.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:05.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:05.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:05.653 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:05.648+0000 7f66c14ed7c0 -1 Falling back to public interface 2024-08-19T11:16:06.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:06.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:06.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:16:06.467 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:16:06.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:06.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:06.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:07.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:07.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:07.934 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:16:07.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:16:07.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:07.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:08.324 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:08.320+0000 7f66c14ed7c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:16:08.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:09.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:09.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:09.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:16:09.483 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:16:09.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:09.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:09.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:10.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:10.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:10.907 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:16:10.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:16:10.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:10.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:11.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:12.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:12.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:12.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:16:12.421 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:16:12.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:12.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:12.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:13.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:13.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:13.855 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:16:13.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:16:13.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:13.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:14.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:15.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:15.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:15.402 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:16:15.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:16:15.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:15.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:15.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:16.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:16.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:16.836 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:16:16.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:16:16.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:16.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:17.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:18.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:18.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:18.391 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:16:18.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:16:18.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:18.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:18.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:19.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:19.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:19.827 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:16:19.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:16:19.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:19.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:20.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:21.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:21.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:21.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:16:21.696 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:16:21.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:21.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:22.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:23.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:23.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:23.129 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:16:23.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:16:23.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:23.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:23.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:24.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:24.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:24.744 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:16:24.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:16:24.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:24.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:25.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:26.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:26.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:26.442 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:16:26.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:16:26.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:26.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:27.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:28.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:28.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:28.390 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:16:28.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:16:28.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:28.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:29.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:30.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:30.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:30.234 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:16:30.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:16:30.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:30.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:30.684 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3468265560,v1:127.0.0.1:6803/3468265560] [v2:127.0.0.1:6804/3468265560,v1:127.0.0.1:6805/3468265560] exists,up 2abecb09-cce2-4be4-a9a9-1bbcc2dff847 2024-08-19T11:16:30.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:16:30.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:16:30.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:16:30.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2024-08-19T11:16:31.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2024-08-19T11:16:31.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2024-08-19T11:16:31.154 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2024-08-19T11:16:31.155 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-08-19T11:16:31.155 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:16:31.155 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:16:31.155 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:16:31.155 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:31.155 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:31.155 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:16:31.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-08-19T11:16:31.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_profile 2024-08-19T11:16:31.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2024-08-19T11:16:31.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2024-08-19T11:16:31.270 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2024-08-19T11:16:31.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2024-08-19T11:16:31.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2024-08-19T11:16:31.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-08-19T11:16:31.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2024-08-19T11:16:31.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2024-08-19T11:16:31.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2024-08-19T11:16:31.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-08-19T11:16:31.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-08-19T11:16:31.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-08-19T11:16:31.272 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-08-19T11:16:31.273 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:16:31.273 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:16:31.273 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:16:31.273 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:31.273 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:31.273 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:16:31.273 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-08-19T11:16:31.273 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-08-19T11:16:31.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2024-08-19T11:16:31.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2024-08-19T11:16:31.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-08-19T11:16:31.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2024-08-19T11:16:31.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2024-08-19T11:16:31.517 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:16:31.517 INFO:tasks.workunit.client.0.smithi130.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '' " 2024-08-19T11:16:31.517 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:16:31.530 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-08-19T11:16:31.531 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-08-19T11:16:31.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-08-19T11:16:31.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-08-19T11:16:31.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-08-19T11:16:33.533 INFO:tasks.workunit.client.0.smithi130.stdout:Check configuration values - Attempt#: 0 2024-08-19T11:16:33.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-08-19T11:16:33.534 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-08-19T11:16:33.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-08-19T11:16:33.956 INFO:tasks.workunit.client.0.smithi130.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-08-19T11:16:33.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-08-19T11:16:33.957 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2024-08-19T11:16:33.957 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-08-19T11:16:33.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-08-19T11:16:33.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-08-19T11:16:33.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2024-08-19T11:16:33.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-08-19T11:16:33.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-08-19T11:16:34.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2024-08-19T11:16:34.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-08-19T11:16:34.371 INFO:tasks.workunit.client.0.smithi130.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-08-19T11:16:34.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2024-08-19T11:16:34.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-08-19T11:16:34.373 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2024-08-19T11:16:34.373 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-08-19T11:16:34.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-08-19T11:16:34.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-08-19T11:16:34.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-08-19T11:16:34.376 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-08-19T11:16:34.376 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:16:34.376 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:16:34.377 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:16:34.377 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:34.377 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:34.377 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:16:34.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-08-19T11:16:34.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-08-19T11:16:34.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2024-08-19T11:16:34.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-08-19T11:16:34.491 INFO:tasks.workunit.client.0.smithi130.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-08-19T11:16:34.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-08-19T11:16:34.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2024-08-19T11:16:34.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2024-08-19T11:16:34.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-08-19T11:16:34.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-08-19T11:16:34.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-08-19T11:16:34.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-08-19T11:16:34.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-08-19T11:16:34.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-08-19T11:16:34.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-08-19T11:16:34.495 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-08-19T11:16:34.495 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:16:34.495 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:16:34.496 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:16:34.496 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:34.496 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:34.496 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:16:34.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-08-19T11:16:34.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-08-19T11:16:34.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2024-08-19T11:16:34.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2024-08-19T11:16:34.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-08-19T11:16:34.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2024-08-19T11:16:34.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2024-08-19T11:16:34.752 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:16:34.752 INFO:tasks.workunit.client.0.smithi130.stdout: "success": "osd_mclock_scheduler_client_res = '' " 2024-08-19T11:16:34.752 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:16:34.765 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-08-19T11:16:34.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-08-19T11:16:34.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-08-19T11:16:34.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-08-19T11:16:34.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-08-19T11:16:36.768 INFO:tasks.workunit.client.0.smithi130.stdout:Check configuration values - Attempt#: 0 2024-08-19T11:16:36.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-08-19T11:16:36.769 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-08-19T11:16:37.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-08-19T11:16:37.225 INFO:tasks.workunit.client.0.smithi130.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-08-19T11:16:37.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-08-19T11:16:37.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2024-08-19T11:16:37.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-08-19T11:16:37.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-08-19T11:16:37.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-08-19T11:16:37.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2024-08-19T11:16:37.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-08-19T11:16:37.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-08-19T11:16:37.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2024-08-19T11:16:37.675 INFO:tasks.workunit.client.0.smithi130.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-08-19T11:16:37.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-08-19T11:16:37.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-08-19T11:16:37.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2024-08-19T11:16:37.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2024-08-19T11:16:37.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-08-19T11:16:37.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-08-19T11:16:37.680 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-08-19T11:16:37.680 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-08-19T11:16:37.680 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-08-19T11:16:37.680 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:16:37.680 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:16:37.681 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:16:37.681 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:37.681 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:37.681 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:16:37.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-08-19T11:16:37.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-08-19T11:16:37.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2024-08-19T11:16:37.786 INFO:tasks.workunit.client.0.smithi130.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-08-19T11:16:37.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-08-19T11:16:37.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2024-08-19T11:16:37.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-08-19T11:16:37.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2024-08-19T11:16:37.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-08-19T11:16:37.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-08-19T11:16:37.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-08-19T11:16:37.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-08-19T11:16:37.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2024-08-19T11:16:37.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:16:37.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:16:37.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:16:37.792 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:16:37.792 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:16:37.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:16:37.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:16:37.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:16:37.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:16:37.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:16:37.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:16:37.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:16:37.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:16:37.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:16:37.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:16:37.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:16:37.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:16:37.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:16:37.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:16:37.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:16:37.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:16:37.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:16:37.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:16:37.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:37.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:37.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:16:37.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:16:37.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:16:37.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-08-19T11:16:37.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:16:37.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:16:37.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:16:37.912 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:16:37.912 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:16:37.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:16:37.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:16:37.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:16:37.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:16:37.914 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:16:37.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:16:37.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:16:37.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:16:37.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:16:37.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:16:37.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:16:37.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:16:37.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:16:37.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:16:37.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:16:37.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:16:37.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:16:37.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:16:37.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:37.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:37.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:16:37.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:16:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:16:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-08-19T11:16:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-08-19T11:16:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-08-19T11:16:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-08-19T11:16:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:16:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:16:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:16:37.923 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:16:37.923 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:16:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:16:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:16:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:16:37.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:16:37.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:16:37.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:16:37.926 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:16:37.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:16:37.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:16:37.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:16:37.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:16:37.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:16:37.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:16:37.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:16:37.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:16:37.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:16:37.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:16:37.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:16:37.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:37.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:37.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:16:37.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:16:37.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:16:37.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-08-19T11:16:37.933 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:16:37.933 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:37.933 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:37.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21434 2024-08-19T11:16:37.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:16:37.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:16:37.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:16:37.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-08-19T11:16:37.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2024-08-19T11:16:37.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2024-08-19T11:16:37.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2024-08-19T11:16:37.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-08-19T11:16:37.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-08-19T11:16:37.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:16:37.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:16:37.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:16:37.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:16:37.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:16:37.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:16:37.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:16:37.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:16:37.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:16:37.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:16:37.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:16:37.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:16:37.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:16:37.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:16:37.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:16:37.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:16:37.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:16:37.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:16:37.941 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:16:37.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:16:37.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:16:37.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:16:37.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:16:37.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:37.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:37.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:16:37.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:16:37.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:16:37.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-08-19T11:16:37.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:16:37.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:37.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:37.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21434 2024-08-19T11:16:37.948 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:16:37.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:16:37.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:16:37.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-08-19T11:16:37.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2024-08-19T11:16:37.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-08-19T11:16:37.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:16:37.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:16:37.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:16:37.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-08-19T11:16:37.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-08-19T11:16:38.327 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:16:38.327 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:16:38.327 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:16:38.328 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:16:38.328 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:38.328 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:38.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:16:38.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:16:38.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:16:38.534 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:16:38.534 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:16:38.534 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:16:38.534 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:16:38.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:16:38.536 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:16:38.536 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:16:38.536 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:16:38.536 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:16:38.536 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:38.536 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:38.536 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-mon.a.asok 2024-08-19T11:16:38.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:16:38.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21434/ceph-mon.a.asok config get fsid 2024-08-19T11:16:38.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:16:38.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:16:38.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:16:38.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:16:38.653 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:16:38.653 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:16:38.653 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:16:38.653 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:16:38.654 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:16:38.654 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:38.654 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:38.654 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-mon.a.asok 2024-08-19T11:16:38.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:16:38.655 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21434/ceph-mon.a.asok config get mon_host 2024-08-19T11:16:38.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-08-19T11:16:38.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-08-19T11:16:38.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:16:38.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:16:38.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:16:38.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-08-19T11:16:38.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:16:39.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:16:39.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:16:39.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:16:39.667 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:16:39.667 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:39.667 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:39.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:16:39.669 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:16:39.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:16:39.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-08-19T11:16:39.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-08-19T11:16:39.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:16:39.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:16:39.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:16:39.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-08-19T11:16:39.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-08-19T11:16:39.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:16:39.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:16:39.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:16:39.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-08-19T11:16:39.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-08-19T11:16:39.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:16:39.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:16:39.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-08-19T11:16:39.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:16:39.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:16:39.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:16:39.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:16:39.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:16:39.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-08-19T11:16:39.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-08-19T11:16:39.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:16:39.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50503591-32e4-41b9-93cf-a9d4fd606cb6 2024-08-19T11:16:39.702 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 50503591-32e4-41b9-93cf-a9d4fd606cb6 2024-08-19T11:16:39.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 50503591-32e4-41b9-93cf-a9d4fd606cb6' 2024-08-19T11:16:39.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:16:39.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXKcNm/W+3KhAAWRwFdgH15IcPy9BDWDNwoQ== 2024-08-19T11:16:39.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXKcNm/W+3KhAAWRwFdgH15IcPy9BDWDNwoQ=="}' 2024-08-19T11:16:39.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50503591-32e4-41b9-93cf-a9d4fd606cb6 -i td/mclock-config/0/new.json 2024-08-19T11:16:39.997 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:16:40.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-08-19T11:16:40.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCXKcNm/W+3KhAAWRwFdgH15IcPy9BDWDNwoQ== --osd-uuid 50503591-32e4-41b9-93cf-a9d4fd606cb6 2024-08-19T11:16:40.046 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:40.043+0000 7ffafc7ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:16:40.057 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:40.051+0000 7ffafc7ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:16:40.059 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:40.055+0000 7ffafc7ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:16:40.059 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:40.055+0000 7ffafc7ea7c0 -1 bdev(0x5573740efc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:16:40.059 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:40.055+0000 7ffafc7ea7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-08-19T11:16:46.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-08-19T11:16:46.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:16:46.048 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:16:46.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:16:46.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:16:46.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:16:46.622 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:16:46.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9e7b848e-8f37-45aa-b11e-19b78f33eada --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-08-19T11:16:46.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:16:46.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:16:46.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:16:46.652 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:46.647+0000 7f27355337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:16:46.654 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:46.651+0000 7f27355337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:16:46.655 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:46.651+0000 7f27355337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:16:47.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:16:47.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:16:47.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:16:47.097 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:16:47.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:16:47.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:16:47.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:47.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:16:47.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:47.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:47.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:48.173 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:48.167+0000 7f27355337c0 -1 Falling back to public interface 2024-08-19T11:16:48.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:48.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:48.518 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:16:48.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:16:48.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:48.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:48.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:49.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:49.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:49.975 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:16:49.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:16:49.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:49.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:50.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:50.944 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:16:50.939+0000 7f27355337c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:16:51.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:51.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:51.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:16:51.456 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:16:51.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:51.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:51.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:52.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:52.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:52.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:16:52.947 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:16:52.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:52.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:53.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:54.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:54.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:54.377 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:16:54.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:16:54.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:54.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:54.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:55.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:55.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:55.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:16:55.837 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:16:55.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:55.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:56.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:57.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:57.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:57.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:16:57.267 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:16:57.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:57.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:57.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:16:58.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:16:58.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:16:58.817 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:16:58.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:16:58.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:16:58.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:16:59.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:00.248 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:17:00.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:00.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:00.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:17:00.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:00.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:00.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:01.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:01.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:01.719 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:17:01.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:17:01.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:01.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:02.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:03.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:03.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:03.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:17:03.337 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:17:03.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:03.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:04.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:05.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:05.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:05.464 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:17:05.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:17:05.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:05.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:06.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:07.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:07.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:07.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:17:07.642 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:17:07.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:07.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:08.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:09.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:09.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:09.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:17:09.496 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:17:09.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:09.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:10.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:11.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:11.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:11.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:17:11.339 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:17:11.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:11.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:12.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:13.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:13.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:13.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:17:13.458 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:17:13.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:13.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:13.932 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/136681752,v1:127.0.0.1:6803/136681752] [v2:127.0.0.1:6804/136681752,v1:127.0.0.1:6805/136681752] exists,up 50503591-32e4-41b9-93cf-a9d4fd606cb6 2024-08-19T11:17:13.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:17:13.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:17:13.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:17:13.933 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-08-19T11:17:13.934 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:17:13.934 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:17:13.934 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:17:13.934 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:13.935 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:17:13.935 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:17:13.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-08-19T11:17:13.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_recovery_max_active 2024-08-19T11:17:14.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2024-08-19T11:17:14.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2024-08-19T11:17:14.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2024-08-19T11:17:14.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-08-19T11:17:14.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-08-19T11:17:16.632 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-08-19T11:17:16.632 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:17:16.632 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:17:16.632 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:17:16.632 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:16.632 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:17:16.633 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:17:16.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-08-19T11:17:16.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_recovery_max_active 2024-08-19T11:17:16.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2024-08-19T11:17:16.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2024-08-19T11:17:16.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-08-19T11:17:17.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-08-19T11:17:17.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-08-19T11:17:19.761 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-08-19T11:17:19.761 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:17:19.761 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:17:19.762 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:17:19.762 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:19.762 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:17:19.762 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21434/ceph-osd.0.asok 2024-08-19T11:17:19.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-08-19T11:17:19.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21434/ceph-osd.0.asok config get osd_recovery_max_active 2024-08-19T11:17:19.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2024-08-19T11:17:19.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2024-08-19T11:17:19.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2024-08-19T11:17:19.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:17:19.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:17:19.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:17:19.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:17:19.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:17:19.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:17:19.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:17:19.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:17:20.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:17:20.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:17:20.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:17:20.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:17:20.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:17:20.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:17:20.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:17:20.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:17:20.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:17:20.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:17:20.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:17:20.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:17:20.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:17:20.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:17:20.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:17:20.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:20.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:17:20.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:17:20.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:17:20.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:17:20.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-08-19T11:17:20.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:17:20.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:17:20.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:17:20.016 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:17:20.016 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:17:20.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:17:20.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:17:20.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:17:20.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:17:20.018 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:17:20.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:17:20.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:17:20.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:17:20.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:17:20.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:17:20.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:17:20.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:17:20.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:17:20.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:17:20.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:17:20.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:17:20.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:17:20.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:17:20.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:20.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:17:20.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:17:20.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:17:20.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:17:20.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2331: main: code=0 2024-08-19T11:17:20.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: teardown td/mclock-config 0 2024-08-19T11:17:20.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-08-19T11:17:20.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-19T11:17:20.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-08-19T11:17:20.026 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:17:20.026 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:17:20.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:17:20.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:17:20.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:17:20.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:17:20.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:17:20.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:17:20.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:17:20.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:17:20.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:17:20.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:17:20.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:17:20.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:17:20.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:17:20.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:17:20.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:17:20.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-19T11:17:20.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-08-19T11:17:20.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:17:20.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:20.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21434 2024-08-19T11:17:20.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21434 2024-08-19T11:17:20.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:17:20.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:17:20.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: main: return 0 2024-08-19T11:17:20.035 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-19T11:17:20.036 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-19T11:17:20.087 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2024-08-19T11:17:20.088 DEBUG:teuthology.orchestra.run.smithi130:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2024-08-19T11:17:20.140 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-19T11:17:20.145 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-19T11:17:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-19T11:17:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-19T11:17:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export PYTHONWARNINGS=ignore 2024-08-19T11:17:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: PYTHONWARNINGS=ignore 2024-08-19T11:17:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: export CEPH_CONF=/dev/null 2024-08-19T11:17:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: CEPH_CONF=/dev/null 2024-08-19T11:17:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: unset CEPH_ARGS 2024-08-19T11:17:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: local code 2024-08-19T11:17:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: run td/network-ping 2024-08-19T11:17:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2024-08-19T11:17:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2024-08-19T11:17:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2024-08-19T11:17:20.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2024-08-19T11:17:20.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2024-08-19T11:17:20.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2024-08-19T11:17:20.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none ' 2024-08-19T11:17:20.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-08-19T11:17:20.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2024-08-19T11:17:20.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2024-08-19T11:17:20.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2024-08-19T11:17:20.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2024-08-19T11:17:20.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-19T11:17:20.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2024-08-19T11:17:20.149 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_network_ping_test2' 2024-08-19T11:17:20.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-08-19T11:17:20.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-08-19T11:17:20.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-08-19T11:17:20.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-08-19T11:17:20.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-08-19T11:17:20.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:17:20.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-08-19T11:17:20.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:17:20.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:17:20.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:17:20.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:17:20.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:17:20.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:17:20.154 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:17:20.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:17:20.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:17:20.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:17:20.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:17:20.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:17:20.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:17:20.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:17:20.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:17:20.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:17:20.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:17:20.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:17:20.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-08-19T11:17:20.158 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:17:20.158 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:20.158 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:17:20.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.35408 2024-08-19T11:17:20.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:17:20.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:17:20.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-08-19T11:17:20.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:17:20.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:20.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:17:20.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.35408 2024-08-19T11:17:20.161 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:17:20.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-19T11:17:20.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-19T11:17:20.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:17:20.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-08-19T11:17:20.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2024-08-19T11:17:20.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2024-08-19T11:17:20.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2024-08-19T11:17:20.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-08-19T11:17:20.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:17:20.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:17:20.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:17:20.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-08-19T11:17:20.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-08-19T11:17:20.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:17:20.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:17:20.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:17:20.628 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:17:20.628 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:20.628 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:17:20.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:17:20.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:17:20.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:17:20.829 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:17:20.829 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:17:20.829 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:17:20.829 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:17:20.829 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:17:20.830 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:17:20.830 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:17:20.830 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:17:20.830 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:17:20.831 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:20.831 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:17:20.831 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-mon.a.asok 2024-08-19T11:17:20.831 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:17:20.831 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.35408/ceph-mon.a.asok config get fsid 2024-08-19T11:17:20.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:17:20.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:17:20.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:17:20.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:17:20.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:17:20.947 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:17:20.947 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:17:20.947 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:17:20.948 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:17:20.948 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:20.949 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:17:20.949 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-mon.a.asok 2024-08-19T11:17:20.949 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:17:20.949 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.35408/ceph-mon.a.asok config get mon_host 2024-08-19T11:17:21.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2024-08-19T11:17:21.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-08-19T11:17:21.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:17:21.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:17:21.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:17:21.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-08-19T11:17:21.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:17:21.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:17:21.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:17:21.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:17:21.969 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:17:21.969 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:21.969 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:17:21.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:17:21.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:17:21.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:17:22.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2024-08-19T11:17:22.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-08-19T11:17:22.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:17:22.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:17:22.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:17:22.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-08-19T11:17:22.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-08-19T11:17:22.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:17:22.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:17:22.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:17:22.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:17:22.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:17:22.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:17:22.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-08-19T11:17:22.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-08-19T11:17:22.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:17:22.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:17:22.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:17:22.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:17:22.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:17:22.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:17:22.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-08-19T11:17:22.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:17:22.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ba82aae7-7d32-46ab-be9c-dfeb7b5ceef8 2024-08-19T11:17:22.010 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 ba82aae7-7d32-46ab-be9c-dfeb7b5ceef8 2024-08-19T11:17:22.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ba82aae7-7d32-46ab-be9c-dfeb7b5ceef8' 2024-08-19T11:17:22.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:17:22.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCKcNm1EGHARAAgAhByvHIziqob/2xYv+Sag== 2024-08-19T11:17:22.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCKcNm1EGHARAAgAhByvHIziqob/2xYv+Sag=="}' 2024-08-19T11:17:22.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ba82aae7-7d32-46ab-be9c-dfeb7b5ceef8 -i td/network-ping/0/new.json 2024-08-19T11:17:22.308 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:17:22.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-08-19T11:17:22.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCKcNm1EGHARAAgAhByvHIziqob/2xYv+Sag== --osd-uuid ba82aae7-7d32-46ab-be9c-dfeb7b5ceef8 2024-08-19T11:17:22.360 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:22.355+0000 7ff4cdbf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:17:22.377 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:22.371+0000 7ff4cdbf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:17:22.379 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:22.375+0000 7ff4cdbf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:17:22.379 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:22.375+0000 7ff4cdbf97c0 -1 bdev(0x563caeb97c00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:17:22.379 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:22.375+0000 7ff4cdbf97c0 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-08-19T11:17:28.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-08-19T11:17:28.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:17:28.337 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:17:28.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:17:28.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:17:28.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:17:28.927 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:17:28.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:17:28.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:17:28.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:17:28.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:17:28.960 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:28.955+0000 7f663aaab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:17:28.962 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:28.959+0000 7f663aaab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:17:28.963 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:28.959+0000 7f663aaab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:17:29.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:17:29.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:17:29.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:17:29.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:17:29.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:17:29.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:29.378 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:17:29.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:17:29.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:29.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:29.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:30.004 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:29.999+0000 7f663aaab7c0 -1 Falling back to public interface 2024-08-19T11:17:30.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:30.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:30.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:17:30.854 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:17:30.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:30.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:31.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:32.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:32.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:32.313 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:17:32.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:17:32.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:32.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:32.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:32.836 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:32.831+0000 7f663aaab7c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:17:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:33.750 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:17:33.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:17:33.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:33.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:34.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:35.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:35.187 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:17:35.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:35.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:17:35.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:35.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:35.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:36.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:36.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:36.717 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:17:36.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:17:36.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:36.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:37.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:38.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:38.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:38.187 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:17:38.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:17:38.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:38.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:38.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:39.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:39.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:39.688 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:17:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:17:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:40.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:41.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:41.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:41.133 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:17:41.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:17:41.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:41.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:41.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:42.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:42.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:42.673 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:17:42.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:17:42.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:42.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:43.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:44.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:44.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:44.110 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:17:44.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:17:44.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:44.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:44.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:45.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:45.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:45.532 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:17:45.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:17:45.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:45.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:46.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:47.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:47.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:47.510 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:17:47.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:17:47.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:47.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:48.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:49.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:49.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:49.286 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:17:49.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:17:49.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:49.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:49.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:50.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:50.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:50.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:17:50.960 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:17:50.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:50.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:51.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:52.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:52.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:52.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:17:52.973 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:17:52.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:52.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:54.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:17:55.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:17:55.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:17:55.177 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:17:55.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:17:55.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:17:55.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:17:55.567 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4005978972,v1:127.0.0.1:6803/4005978972] [v2:127.0.0.1:6804/4005978972,v1:127.0.0.1:6805/4005978972] exists,up ba82aae7-7d32-46ab-be9c-dfeb7b5ceef8 2024-08-19T11:17:55.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:17:55.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:17:55.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:17:55.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2024-08-19T11:17:55.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-08-19T11:17:55.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:17:55.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T11:17:55.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:17:55.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-08-19T11:17:55.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-08-19T11:17:55.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:17:55.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:17:55.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:17:55.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-08-19T11:17:55.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-08-19T11:17:55.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:17:55.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:17:55.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-08-19T11:17:55.572 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:17:55.572 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:17:55.572 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:17:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:17:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:17:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:17:55.573 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:17:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:17:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:17:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:17:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:17:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-08-19T11:17:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-08-19T11:17:55.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:17:55.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:17:55.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:17:55.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:17:55.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:17:55.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:17:55.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-08-19T11:17:55.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:17:55.576 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 736ce174-f05b-4d09-93e1-fbfe889c8053 2024-08-19T11:17:55.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=736ce174-f05b-4d09-93e1-fbfe889c8053 2024-08-19T11:17:55.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 736ce174-f05b-4d09-93e1-fbfe889c8053' 2024-08-19T11:17:55.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:17:55.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDjKcNmufq3IxAAewp059PBBVRELtowWL3QxQ== 2024-08-19T11:17:55.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDjKcNmufq3IxAAewp059PBBVRELtowWL3QxQ=="}' 2024-08-19T11:17:55.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 736ce174-f05b-4d09-93e1-fbfe889c8053 -i td/network-ping/1/new.json 2024-08-19T11:17:56.093 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:17:56.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-08-19T11:17:56.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDjKcNmufq3IxAAewp059PBBVRELtowWL3QxQ== --osd-uuid 736ce174-f05b-4d09-93e1-fbfe889c8053 2024-08-19T11:17:56.145 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:56.138+0000 7f89ae5117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:17:56.147 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:56.142+0000 7f89ae5117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:17:56.148 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:56.142+0000 7f89ae5117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:17:56.149 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:56.142+0000 7f89ae5117c0 -1 bdev(0x55bb73dd7c00 td/network-ping/1/block) open stat got: (1) Operation not permitted 2024-08-19T11:17:56.149 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:17:56.142+0000 7f89ae5117c0 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-08-19T11:18:02.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-08-19T11:18:02.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:18:02.107 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-08-19T11:18:02.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T11:18:02.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:18:02.672 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-08-19T11:18:02.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T11:18:02.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:18:02.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:18:02.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:18:02.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:18:02.706 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:02.702+0000 7f3781b647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:18:02.718 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:02.714+0000 7f3781b647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:18:02.719 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:02.714+0000 7f3781b647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:18:03.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T11:18:03.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:18:03.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T11:18:03.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:18:03.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:18:03.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:03.147 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:18:03.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:18:03.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:03.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:03.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:03.920 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:03.914+0000 7f3781b647c0 -1 Falling back to public interface 2024-08-19T11:18:04.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:04.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:04.624 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:18:04.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:18:04.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:04.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:05.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:06.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:06.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:06.036 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:18:06.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:18:06.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:06.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:06.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:06.682 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:06.678+0000 7f3781b647c0 -1 osd.1 0 log_to_monitors true 2024-08-19T11:18:07.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:07.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:07.455 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:18:07.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:18:07.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:07.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:08.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:09.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:09.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:09.024 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:18:09.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:18:09.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:09.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:09.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:10.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:10.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:10.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:18:10.466 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:18:10.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:10.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:10.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:11.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:11.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:11.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:18:11.959 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:18:11.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:11.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:12.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:13.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:13.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:13.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:18:13.378 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:18:13.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:13.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:13.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:14.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:14.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:14.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:18:14.807 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:18:14.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:14.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:15.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:16.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:16.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:16.320 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:18:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:18:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:16.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:17.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:17.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:17.957 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:18:17.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:18:17.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:17.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:18.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:19.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:19.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:19.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:18:19.488 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:18:19.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:19.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:20.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:21.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:21.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:21.267 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:18:21.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:18:21.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:21.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:22.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:23.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:23.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:23.330 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:18:23.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:18:23.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:23.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:24.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:25.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:25.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:25.304 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:18:25.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:18:25.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:25.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:26.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:27.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:27.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:27.409 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:18:27.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:18:27.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:27.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:28.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:29.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:29.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:29.416 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:18:29.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:18:29.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:29.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:18:29.848 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1965844507,v1:127.0.0.1:6811/1965844507] [v2:127.0.0.1:6812/1965844507,v1:127.0.0.1:6813/1965844507] exists,up 736ce174-f05b-4d09-93e1-fbfe889c8053 2024-08-19T11:18:29.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:18:29.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:18:29.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:18:29.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2024-08-19T11:18:29.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-08-19T11:18:29.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:18:29.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T11:18:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:18:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-08-19T11:18:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-08-19T11:18:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:18:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:18:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:18:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-08-19T11:18:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-08-19T11:18:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:18:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:18:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-08-19T11:18:29.855 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:18:29.855 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:18:29.855 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:18:29.857 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:18:29.857 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:18:29.857 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:18:29.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:18:29.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:18:29.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:18:29.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:18:29.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:18:29.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-08-19T11:18:29.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-08-19T11:18:29.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:18:29.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:18:29.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:18:29.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:18:29.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:18:29.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:18:29.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-08-19T11:18:29.858 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:18:29.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=182c1e0b-dee9-4dd5-8f78-406742bf91fd 2024-08-19T11:18:29.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 182c1e0b-dee9-4dd5-8f78-406742bf91fd' 2024-08-19T11:18:29.859 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 182c1e0b-dee9-4dd5-8f78-406742bf91fd 2024-08-19T11:18:29.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:18:29.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAFKsNmyZNRNBAAF1ZNHKaY2gDXhpZkevWyfQ== 2024-08-19T11:18:29.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAFKsNmyZNRNBAAF1ZNHKaY2gDXhpZkevWyfQ=="}' 2024-08-19T11:18:29.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 182c1e0b-dee9-4dd5-8f78-406742bf91fd -i td/network-ping/2/new.json 2024-08-19T11:18:30.380 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:18:30.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-08-19T11:18:30.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAFKsNmyZNRNBAAF1ZNHKaY2gDXhpZkevWyfQ== --osd-uuid 182c1e0b-dee9-4dd5-8f78-406742bf91fd 2024-08-19T11:18:30.432 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:30.426+0000 7fe4fd3877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:18:30.434 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:30.430+0000 7fe4fd3877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:18:30.435 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:30.430+0000 7fe4fd3877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:18:30.436 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:30.430+0000 7fe4fd3877c0 -1 bdev(0x558409b51c00 td/network-ping/2/block) open stat got: (1) Operation not permitted 2024-08-19T11:18:30.436 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:30.430+0000 7fe4fd3877c0 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-08-19T11:18:36.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-08-19T11:18:36.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:18:36.298 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-08-19T11:18:36.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T11:18:36.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:18:36.877 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-08-19T11:18:36.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T11:18:36.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:18:36.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:18:36.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:18:36.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:18:36.904 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:36.902+0000 7f7de1e757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:18:36.906 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:36.906+0000 7f7de1e757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:18:36.907 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:36.906+0000 7f7de1e757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:18:37.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T11:18:37.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:18:37.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T11:18:37.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:18:37.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:18:37.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:37.301 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:18:37.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:18:37.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:37.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:37.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:38.404 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:38.402+0000 7f7de1e757c0 -1 Falling back to public interface 2024-08-19T11:18:38.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:38.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:38.738 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:18:38.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:18:38.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:38.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:39.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:40.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:40.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:40.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:18:40.178 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:18:40.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:40.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:40.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:41.354 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:18:41.354+0000 7f7de1e757c0 -1 osd.2 0 log_to_monitors true 2024-08-19T11:18:41.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:41.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:41.704 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:18:41.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:18:41.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:41.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:42.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:43.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:43.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:43.439 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:18:43.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:18:43.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:43.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:43.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:44.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:44.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:44.909 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:18:44.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:18:44.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:44.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:45.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:46.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:46.514 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:18:46.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:46.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:18:46.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:46.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:46.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:47.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:47.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:47.950 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:18:47.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:18:47.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:47.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:48.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:49.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:49.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:49.478 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:18:49.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:18:49.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:49.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:50.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:51.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:51.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:51.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:18:51.139 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:18:51.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:51.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:51.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:52.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:52.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:52.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:18:52.586 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:18:52.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:52.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:53.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:54.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:54.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:54.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:18:54.121 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:18:54.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:54.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:54.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:18:55.911 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:18:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:55.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:56.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:57.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:57.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:57.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:18:57.670 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:18:57.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:57.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:18:58.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:18:59.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:18:59.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:18:59.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:18:59.475 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:18:59.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:18:59.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:19:00.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:19:01.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:19:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:19:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:19:01.467 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:19:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:19:01.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:19:02.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:19:03.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:19:03.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:19:03.482 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:19:03.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:19:03.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:19:03.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:19:04.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:19:05.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:19:05.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:19:05.043 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:19:05.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:19:05.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:19:05.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:19:05.463 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2578760414,v1:127.0.0.1:6819/2578760414] [v2:127.0.0.1:6820/2578760414,v1:127.0.0.1:6821/2578760414] exists,up 182c1e0b-dee9-4dd5-8f78-406742bf91fd 2024-08-19T11:19:05.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:19:05.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:19:05.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:19:05.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2024-08-19T11:19:10.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2024-08-19T11:19:10.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-08-19T11:19:10.999 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'foo' already exists 2024-08-19T11:19:11.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:19:12.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-08-19T11:19:12.236 INFO:tasks.workunit.client.0.smithi130.stdout:hints = 1 2024-08-19T11:19:12.236 INFO:tasks.workunit.client.0.smithi130.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-08-19T11:19:12.236 INFO:tasks.workunit.client.0.smithi130.stdout:Object prefix: benchmark_data_smithi130_40885 2024-08-19T11:19:12.236 INFO:tasks.workunit.client.0.smithi130.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T11:19:12.236 INFO:tasks.workunit.client.0.smithi130.stdout: 0 0 0 0 0 0 - 0 2024-08-19T11:19:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: 1 16 77 61 0.238256 0.238281 0.252851 0.225977 2024-08-19T11:19:14.236 INFO:tasks.workunit.client.0.smithi130.stdout: 2 16 133 117 0.228491 0.21875 0.217075 0.24808 2024-08-19T11:19:15.236 INFO:tasks.workunit.client.0.smithi130.stdout: 3 16 197 181 0.235649 0.25 0.248747 0.244199 2024-08-19T11:19:16.236 INFO:tasks.workunit.client.0.smithi130.stdout: 4 16 276 260 0.253877 0.308594 0.253057 0.245352 2024-08-19T11:19:17.237 INFO:tasks.workunit.client.0.smithi130.stdout: 5 16 340 324 0.253094 0.25 0.252459 0.245724 2024-08-19T11:19:18.237 INFO:tasks.workunit.client.0.smithi130.stdout: 6 16 404 388 0.252572 0.25 0.239997 0.245114 2024-08-19T11:19:19.237 INFO:tasks.workunit.client.0.smithi130.stdout: 7 16 468 452 0.252199 0.25 0.257126 0.245184 2024-08-19T11:19:20.237 INFO:tasks.workunit.client.0.smithi130.stdout: 8 16 532 516 0.251922 0.25 0.231847 0.244155 2024-08-19T11:19:21.237 INFO:tasks.workunit.client.0.smithi130.stdout: 9 16 609 593 0.257342 0.300781 0.169289 0.241858 2024-08-19T11:19:22.237 INFO:tasks.workunit.client.0.smithi130.stdout: 10 16 693 677 0.264416 0.328125 0.201785 0.235556 2024-08-19T11:19:23.237 INFO:tasks.workunit.client.0.smithi130.stdout:Total time run: 10.1412 2024-08-19T11:19:23.238 INFO:tasks.workunit.client.0.smithi130.stdout:Total writes made: 693 2024-08-19T11:19:23.238 INFO:tasks.workunit.client.0.smithi130.stdout:Write size: 4096 2024-08-19T11:19:23.238 INFO:tasks.workunit.client.0.smithi130.stdout:Object size: 4096 2024-08-19T11:19:23.238 INFO:tasks.workunit.client.0.smithi130.stdout:Bandwidth (MB/sec): 0.266934 2024-08-19T11:19:23.238 INFO:tasks.workunit.client.0.smithi130.stdout:Stddev Bandwidth: 0.0351828 2024-08-19T11:19:23.238 INFO:tasks.workunit.client.0.smithi130.stdout:Max bandwidth (MB/sec): 0.328125 2024-08-19T11:19:23.238 INFO:tasks.workunit.client.0.smithi130.stdout:Min bandwidth (MB/sec): 0.21875 2024-08-19T11:19:23.238 INFO:tasks.workunit.client.0.smithi130.stdout:Average IOPS: 68 2024-08-19T11:19:23.238 INFO:tasks.workunit.client.0.smithi130.stdout:Stddev IOPS: 9.00679 2024-08-19T11:19:23.238 INFO:tasks.workunit.client.0.smithi130.stdout:Max IOPS: 84 2024-08-19T11:19:23.238 INFO:tasks.workunit.client.0.smithi130.stdout:Min IOPS: 56 2024-08-19T11:19:23.238 INFO:tasks.workunit.client.0.smithi130.stdout:Average Latency(s): 0.234003 2024-08-19T11:19:23.239 INFO:tasks.workunit.client.0.smithi130.stdout:Stddev Latency(s): 0.0397362 2024-08-19T11:19:23.239 INFO:tasks.workunit.client.0.smithi130.stdout:Max latency(s): 0.453998 2024-08-19T11:19:23.239 INFO:tasks.workunit.client.0.smithi130.stdout:Min latency(s): 0.0936576 2024-08-19T11:19:23.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2024-08-19T11:19:33.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2024-08-19T11:19:33.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:19:33.335 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:19:33.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-19T11:19:33.760 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-08-19T11:19:33.760 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-08-19T11:19:33.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:19:33.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:19:33.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:19:33.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836509 2024-08-19T11:19:33.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836509 2024-08-19T11:19:33.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836509' 2024-08-19T11:19:33.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:19:33.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T11:19:34.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672979 2024-08-19T11:19:34.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672979 2024-08-19T11:19:34.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836509 1-42949672979' 2024-08-19T11:19:34.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:19:34.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T11:19:34.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509453 2024-08-19T11:19:34.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509453 2024-08-19T11:19:34.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836509 1-42949672979 2-64424509453' 2024-08-19T11:19:34.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:19:34.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836509 2024-08-19T11:19:34.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:19:34.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:19:34.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836509 2024-08-19T11:19:34.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:19:34.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836509 2024-08-19T11:19:34.214 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836509 2024-08-19T11:19:34.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-08-19T11:19:34.214 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:19:34.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836507 -lt 21474836509 2024-08-19T11:19:34.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:19:35.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:19:35.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:19:36.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836509 -lt 21474836509 2024-08-19T11:19:36.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:19:36.137 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672979 2024-08-19T11:19:36.137 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:19:36.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-19T11:19:36.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672979 2024-08-19T11:19:36.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:19:36.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672979 2024-08-19T11:19:36.140 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672979 2024-08-19T11:19:36.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-08-19T11:19:36.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T11:19:36.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672979 -lt 42949672979 2024-08-19T11:19:36.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:19:36.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509453 2024-08-19T11:19:36.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:19:36.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-19T11:19:36.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509453 2024-08-19T11:19:36.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:19:36.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509453 2024-08-19T11:19:36.561 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509453 2024-08-19T11:19:36.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-08-19T11:19:36.561 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T11:19:36.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509454 -lt 64424509453 2024-08-19T11:19:36.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:19:36.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2024-08-19T11:19:36.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:19:36.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:19:36.977 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:19:36.978 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:19:36.978 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:19:36.978 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-osd.0.asok 2024-08-19T11:19:36.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:19:36.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-osd.0.asok dump_osd_network 2024-08-19T11:19:37.078 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:19:37.079 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 1000, 2024-08-19T11:19:37.079 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [] 2024-08-19T11:19:37.079 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:19:37.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:19:37.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2024-08-19T11:19:37.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2024-08-19T11:19:37.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:19:37.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2024-08-19T11:19:37.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2024-08-19T11:19:37.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:19:37.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2024-08-19T11:19:37.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-08-19T11:19:37.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-08-19T11:19:37.166 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:19:37.166 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:19:37.166 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:19:37.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-mgr.x.asok 2024-08-19T11:19:37.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:19:37.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-mgr.x.asok dump_osd_network 2024-08-19T11:19:37.261 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:19:37.262 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 1000, 2024-08-19T11:19:37.262 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [] 2024-08-19T11:19:37.262 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:19:37.274 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:19:37.274 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2024-08-19T11:19:37.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2024-08-19T11:19:37.301 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:19:37.301 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2024-08-19T11:19:37.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2024-08-19T11:19:37.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:19:37.329 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2024-08-19T11:19:37.329 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:19:37.329 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:19:37.329 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:19:37.329 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:19:37.330 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:19:37.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-osd.0.asok 2024-08-19T11:19:37.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:19:37.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-osd.0.asok dump_osd_network 0 2024-08-19T11:19:37.437 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:19:37.438 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 0, 2024-08-19T11:19:37.438 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [ 2024-08-19T11:19:37.438 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.438 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:34 2024", 2024-08-19T11:19:37.438 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.438 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:19:37.438 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:19:37.438 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:19:37.438 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.438 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.549, 2024-08-19T11:19:37.438 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.620, 2024-08-19T11:19:37.439 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.522 2024-08-19T11:19:37.439 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.439 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.439 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.661, 2024-08-19T11:19:37.439 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 2.595, 2024-08-19T11:19:37.439 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 2.595 2024-08-19T11:19:37.439 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.439 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.439 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.661, 2024-08-19T11:19:37.439 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 2.595, 2024-08-19T11:19:37.439 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 2.595 2024-08-19T11:19:37.439 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.439 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.391 2024-08-19T11:19:37.440 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.440 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.440 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:36 2024", 2024-08-19T11:19:37.440 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.440 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:19:37.440 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:19:37.440 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:19:37.440 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.440 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.495, 2024-08-19T11:19:37.440 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.609, 2024-08-19T11:19:37.440 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.591 2024-08-19T11:19:37.440 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.440 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.441 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.568, 2024-08-19T11:19:37.441 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 1.227, 2024-08-19T11:19:37.441 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 1.227 2024-08-19T11:19:37.441 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.441 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.441 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.568, 2024-08-19T11:19:37.441 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 1.227, 2024-08-19T11:19:37.441 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 1.227 2024-08-19T11:19:37.441 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.441 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.545 2024-08-19T11:19:37.441 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.441 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.441 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:36 2024", 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.479, 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.603, 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.581 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.603, 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 2.692, 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 2.692 2024-08-19T11:19:37.442 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.443 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.443 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.603, 2024-08-19T11:19:37.443 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 2.692, 2024-08-19T11:19:37.443 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 2.692 2024-08-19T11:19:37.443 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.443 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.603 2024-08-19T11:19:37.443 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.443 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.443 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:34 2024", 2024-08-19T11:19:37.443 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.443 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:19:37.443 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:19:37.444 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:19:37.444 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.444 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.482, 2024-08-19T11:19:37.444 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.533, 2024-08-19T11:19:37.444 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.494 2024-08-19T11:19:37.444 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.444 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.444 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.615, 2024-08-19T11:19:37.444 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.657, 2024-08-19T11:19:37.444 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.657 2024-08-19T11:19:37.444 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.444 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.444 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.615, 2024-08-19T11:19:37.445 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.657, 2024-08-19T11:19:37.445 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.657 2024-08-19T11:19:37.445 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.445 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.479 2024-08-19T11:19:37.445 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-08-19T11:19:37.445 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-08-19T11:19:37.445 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:19:37.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:19:37.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2024-08-19T11:19:37.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2024-08-19T11:19:37.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2024-08-19T11:19:37.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:19:37.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2024-08-19T11:19:37.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:19:37.523 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2024-08-19T11:19:37.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-08-19T11:19:37.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-08-19T11:19:37.524 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:19:37.524 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:19:37.524 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:19:37.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-mgr.x.asok 2024-08-19T11:19:37.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:19:37.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-mgr.x.asok dump_osd_network 0 2024-08-19T11:19:37.620 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:19:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 0, 2024-08-19T11:19:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [ 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:31 2024", 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.571, 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.626, 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.596 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.517, 2024-08-19T11:19:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.516, 2024-08-19T11:19:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.516 2024-08-19T11:19:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.607, 2024-08-19T11:19:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 1.227, 2024-08-19T11:19:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 1.227 2024-08-19T11:19:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.568 2024-08-19T11:19:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:31 2024", 2024-08-19T11:19:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.538, 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.621, 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.587 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.475, 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.457, 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.457 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.565, 2024-08-19T11:19:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 2.692, 2024-08-19T11:19:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 2.692 2024-08-19T11:19:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.511 2024-08-19T11:19:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:34 2024", 2024-08-19T11:19:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:19:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:19:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:19:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.549, 2024-08-19T11:19:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.620, 2024-08-19T11:19:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.522 2024-08-19T11:19:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.488, 2024-08-19T11:19:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.435, 2024-08-19T11:19:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.235 2024-08-19T11:19:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.661, 2024-08-19T11:19:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 2.595, 2024-08-19T11:19:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 2.595 2024-08-19T11:19:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.597 2024-08-19T11:19:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:30 2024", 2024-08-19T11:19:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 1, 2024-08-19T11:19:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:19:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:19:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.594, 2024-08-19T11:19:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.593, 2024-08-19T11:19:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.581 2024-08-19T11:19:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.538, 2024-08-19T11:19:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.452, 2024-08-19T11:19:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.452 2024-08-19T11:19:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.641, 2024-08-19T11:19:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.845, 2024-08-19T11:19:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.845 2024-08-19T11:19:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.484 2024-08-19T11:19:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:34 2024", 2024-08-19T11:19:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 2, 2024-08-19T11:19:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:19:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:19:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.535, 2024-08-19T11:19:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.573, 2024-08-19T11:19:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.591 2024-08-19T11:19:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.399, 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.399, 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.399 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.604, 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.686, 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.689 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.399 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:34 2024", 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 2, 2024-08-19T11:19:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:19:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:19:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.498, 2024-08-19T11:19:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.535, 2024-08-19T11:19:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.556 2024-08-19T11:19:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.481, 2024-08-19T11:19:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.430, 2024-08-19T11:19:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.430 2024-08-19T11:19:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.519, 2024-08-19T11:19:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.892, 2024-08-19T11:19:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.892 2024-08-19T11:19:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.508 2024-08-19T11:19:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:30 2024", 2024-08-19T11:19:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 1, 2024-08-19T11:19:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:19:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:19:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.547, 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.537, 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.523 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.459, 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.418, 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.418 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.605, 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.709, 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.709 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.289 2024-08-19T11:19:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:34 2024", 2024-08-19T11:19:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 1, 2024-08-19T11:19:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 0, 2024-08-19T11:19:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:19:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.505, 2024-08-19T11:19:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.539, 2024-08-19T11:19:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.524 2024-08-19T11:19:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.318, 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.318, 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.233 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.599, 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.742, 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.772 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.318 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:34 2024", 2024-08-19T11:19:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 2, 2024-08-19T11:19:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 0, 2024-08-19T11:19:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:19:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.496, 2024-08-19T11:19:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.534, 2024-08-19T11:19:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.537 2024-08-19T11:19:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.318, 2024-08-19T11:19:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.318, 2024-08-19T11:19:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.290 2024-08-19T11:19:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.556, 2024-08-19T11:19:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.675, 2024-08-19T11:19:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.675 2024-08-19T11:19:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.318 2024-08-19T11:19:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:34 2024", 2024-08-19T11:19:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.482, 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.533, 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.494 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.345, 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.345, 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.265 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.615, 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.657, 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.657 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.435 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:34 2024", 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 2, 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 0, 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.424, 2024-08-19T11:19:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.508, 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.495 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.346, 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.346, 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.346 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.463, 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.673, 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.673 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.346 2024-08-19T11:19:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:34 2024", 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 1, 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 0, 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.412, 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.499, 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.475 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.261, 2024-08-19T11:19:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.261, 2024-08-19T11:19:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.261 2024-08-19T11:19:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:19:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.454, 2024-08-19T11:19:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.672, 2024-08-19T11:19:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.672 2024-08-19T11:19:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:19:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.261 2024-08-19T11:19:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-08-19T11:19:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-08-19T11:19:37.641 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:19:37.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:19:37.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2024-08-19T11:19:37.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2024-08-19T11:19:37.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:19:37.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2024-08-19T11:19:37.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2024-08-19T11:19:37.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2024-08-19T11:19:57.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2024-08-19T11:19:57.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:19:57.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:19:58.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-19T11:19:58.160 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-08-19T11:19:58.160 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-08-19T11:19:58.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:19:58.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:19:58.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:19:58.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836516 2024-08-19T11:19:58.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836516 2024-08-19T11:19:58.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836516' 2024-08-19T11:19:58.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:19:58.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T11:19:58.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672986 2024-08-19T11:19:58.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672986 2024-08-19T11:19:58.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836516 1-42949672986' 2024-08-19T11:19:58.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:19:58.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T11:19:58.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509460 2024-08-19T11:19:58.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509460 2024-08-19T11:19:58.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836516 1-42949672986 2-64424509460' 2024-08-19T11:19:58.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:19:58.647 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836516 2024-08-19T11:19:58.647 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:19:58.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:19:58.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836516 2024-08-19T11:19:58.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:19:58.651 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836516 2024-08-19T11:19:58.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836516 2024-08-19T11:19:58.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-08-19T11:19:58.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:19:59.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836516 -lt 21474836516 2024-08-19T11:19:59.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:19:59.096 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672986 2024-08-19T11:19:59.096 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:19:59.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-19T11:19:59.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672986 2024-08-19T11:19:59.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:19:59.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672986 2024-08-19T11:19:59.099 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672986 2024-08-19T11:19:59.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-08-19T11:19:59.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T11:19:59.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672986 -lt 42949672986 2024-08-19T11:19:59.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:19:59.572 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509460 2024-08-19T11:19:59.572 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:19:59.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-19T11:19:59.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509460 2024-08-19T11:19:59.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:19:59.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509460 2024-08-19T11:19:59.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-08-19T11:19:59.576 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509460 2024-08-19T11:19:59.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T11:20:00.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509460 -lt 64424509460 2024-08-19T11:20:00.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:20:00.040 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2024-08-19T11:20:00.040 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:20:00.040 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:20:00.041 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:20:00.041 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:00.041 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:00.041 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-osd.0.asok 2024-08-19T11:20:00.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:20:00.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-osd.0.asok dump_osd_network 2024-08-19T11:20:00.150 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:20:00.150 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 1000, 2024-08-19T11:20:00.150 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [] 2024-08-19T11:20:00.150 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:20:00.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:00.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2024-08-19T11:20:00.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2024-08-19T11:20:00.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2024-08-19T11:20:00.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:00.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2024-08-19T11:20:00.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:20:00.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2024-08-19T11:20:00.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-08-19T11:20:00.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-08-19T11:20:00.238 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:20:00.238 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:00.238 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:00.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-mgr.x.asok 2024-08-19T11:20:00.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:20:00.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-mgr.x.asok dump_osd_network 2024-08-19T11:20:00.335 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:20:00.336 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 1000, 2024-08-19T11:20:00.336 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [] 2024-08-19T11:20:00.336 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:20:00.349 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:00.349 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2024-08-19T11:20:00.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2024-08-19T11:20:00.385 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:00.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2024-08-19T11:20:00.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2024-08-19T11:20:00.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:20:00.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2024-08-19T11:20:00.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:20:00.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:20:00.424 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:20:00.424 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:00.424 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:00.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-osd.0.asok 2024-08-19T11:20:00.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:20:00.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-osd.0.asok dump_osd_network 0 2024-08-19T11:20:00.535 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:20:00.535 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 0, 2024-08-19T11:20:00.535 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [ 2024-08-19T11:20:00.535 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.535 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:56 2024", 2024-08-19T11:20:00.535 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.535 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:00.535 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:20:00.535 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:00.535 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.535 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.592, 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.568, 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.592 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.651, 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.738, 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 1.227 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.651, 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.738, 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 1.227 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.519 2024-08-19T11:20:00.536 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.537 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.537 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:56 2024", 2024-08-19T11:20:00.537 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.537 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:00.537 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:20:00.537 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:00.537 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.537 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.543, 2024-08-19T11:20:00.537 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.541, 2024-08-19T11:20:00.537 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.577 2024-08-19T11:20:00.537 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.537 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.537 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.619, 2024-08-19T11:20:00.538 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.644, 2024-08-19T11:20:00.538 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 2.692 2024-08-19T11:20:00.538 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.538 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.538 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.619, 2024-08-19T11:20:00.538 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.644, 2024-08-19T11:20:00.538 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 2.692 2024-08-19T11:20:00.538 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.538 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.374 2024-08-19T11:20:00.538 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.538 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.538 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:59 2024", 2024-08-19T11:20:00.538 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.539 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:00.539 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:20:00.539 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:00.539 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.539 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.482, 2024-08-19T11:20:00.539 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.476, 2024-08-19T11:20:00.539 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.532 2024-08-19T11:20:00.539 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.539 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.539 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.587, 2024-08-19T11:20:00.539 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.650, 2024-08-19T11:20:00.539 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 2.595 2024-08-19T11:20:00.539 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.587, 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.650, 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 2.595 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.488 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:59 2024", 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:00.540 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.469, 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.525, 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.520 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.573, 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.686, 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.686 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.573, 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.686, 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.686 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.541 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.454 2024-08-19T11:20:00.542 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-08-19T11:20:00.542 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-08-19T11:20:00.542 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:20:00.549 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:00.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2024-08-19T11:20:00.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2024-08-19T11:20:00.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:00.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2024-08-19T11:20:00.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2024-08-19T11:20:00.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:20:00.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2024-08-19T11:20:00.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-08-19T11:20:00.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-08-19T11:20:00.624 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:20:00.624 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:00.624 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:00.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-mgr.x.asok 2024-08-19T11:20:00.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:20:00.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-mgr.x.asok dump_osd_network 0 2024-08-19T11:20:00.719 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:20:00.719 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 0, 2024-08-19T11:20:00.719 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [ 2024-08-19T11:20:00.719 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.719 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:56 2024", 2024-08-19T11:20:00.719 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.719 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:00.719 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:20:00.719 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:00.720 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.720 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.592, 2024-08-19T11:20:00.720 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.568, 2024-08-19T11:20:00.720 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.592 2024-08-19T11:20:00.720 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.720 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.720 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.538, 2024-08-19T11:20:00.720 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.396, 2024-08-19T11:20:00.720 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.396 2024-08-19T11:20:00.720 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.720 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.720 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.651, 2024-08-19T11:20:00.720 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.738, 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 1.227 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.365 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:54 2024", 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 2, 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.554, 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.552, 2024-08-19T11:20:00.721 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.579 2024-08-19T11:20:00.722 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.722 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.722 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.518, 2024-08-19T11:20:00.722 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.303, 2024-08-19T11:20:00.722 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.303 2024-08-19T11:20:00.722 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.722 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.722 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.618, 2024-08-19T11:20:00.722 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.646, 2024-08-19T11:20:00.722 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.689 2024-08-19T11:20:00.722 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.722 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.330 2024-08-19T11:20:00.722 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:56 2024", 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.543, 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.541, 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.577 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.492, 2024-08-19T11:20:00.723 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.312, 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.312 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.619, 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.644, 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 2.692 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.381 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:55 2024", 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 1, 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:00.724 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.725 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.565, 2024-08-19T11:20:00.725 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.550, 2024-08-19T11:20:00.725 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.573 2024-08-19T11:20:00.725 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.725 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.725 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.497, 2024-08-19T11:20:00.725 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.295, 2024-08-19T11:20:00.725 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.295 2024-08-19T11:20:00.725 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.725 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.725 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.639, 2024-08-19T11:20:00.725 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.687, 2024-08-19T11:20:00.725 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.845 2024-08-19T11:20:00.726 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.726 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.643 2024-08-19T11:20:00.726 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.726 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.726 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:54 2024", 2024-08-19T11:20:00.726 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.726 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 1, 2024-08-19T11:20:00.726 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 0, 2024-08-19T11:20:00.726 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:00.726 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.550, 2024-08-19T11:20:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.525, 2024-08-19T11:20:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.515 2024-08-19T11:20:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.524, 2024-08-19T11:20:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.244, 2024-08-19T11:20:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.233 2024-08-19T11:20:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.591, 2024-08-19T11:20:00.727 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.670, 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.742 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.539 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:54 2024", 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 2, 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.530, 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.518, 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.544 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.728 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.430, 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.261, 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.261 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.603, 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.603, 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.892 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.355 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:54 2024", 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.729 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.543, 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.528, 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.517 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.487, 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.320, 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.265 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.686, 2024-08-19T11:20:00.730 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.686, 2024-08-19T11:20:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.686 2024-08-19T11:20:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.331 2024-08-19T11:20:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:55 2024", 2024-08-19T11:20:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 1, 2024-08-19T11:20:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:20:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.731 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.533, 2024-08-19T11:20:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.515, 2024-08-19T11:20:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.523 2024-08-19T11:20:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.466, 2024-08-19T11:20:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.289, 2024-08-19T11:20:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.289 2024-08-19T11:20:00.732 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.605, 2024-08-19T11:20:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.734, 2024-08-19T11:20:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.734 2024-08-19T11:20:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.462 2024-08-19T11:20:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:54 2024", 2024-08-19T11:20:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 2, 2024-08-19T11:20:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 0, 2024-08-19T11:20:00.733 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.500, 2024-08-19T11:20:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.511, 2024-08-19T11:20:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.531 2024-08-19T11:20:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.478, 2024-08-19T11:20:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.315, 2024-08-19T11:20:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.290 2024-08-19T11:20:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.526, 2024-08-19T11:20:00.734 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.619, 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.675 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.496 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:54 2024", 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.482, 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.489, 2024-08-19T11:20:00.735 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.528 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.424, 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.301, 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.235 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.552, 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.661, 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 2.595 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.313 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.736 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:54 2024", 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 2, 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 0, 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.509, 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.473, 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.494 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.415, 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.329, 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.329 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.737 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.605, 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.646, 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.673 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.414 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:19:54 2024", 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 1, 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 0, 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.495, 2024-08-19T11:20:00.738 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.475, 2024-08-19T11:20:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.485 2024-08-19T11:20:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.434, 2024-08-19T11:20:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.261, 2024-08-19T11:20:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.261 2024-08-19T11:20:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.562, 2024-08-19T11:20:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.679, 2024-08-19T11:20:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.679 2024-08-19T11:20:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:00.739 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.398 2024-08-19T11:20:00.740 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-08-19T11:20:00.740 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-08-19T11:20:00.740 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:20:00.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2024-08-19T11:20:00.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:00.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2024-08-19T11:20:00.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:00.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2024-08-19T11:20:00.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2024-08-19T11:20:00.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2024-08-19T11:20:50.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2024-08-19T11:20:50.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:20:50.797 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:20:51.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-19T11:20:51.232 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-08-19T11:20:51.232 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-08-19T11:20:51.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:20:51.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:20:51.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:20:51.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836529 2024-08-19T11:20:51.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836529 2024-08-19T11:20:51.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836529' 2024-08-19T11:20:51.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:20:51.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T11:20:51.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672999 2024-08-19T11:20:51.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672999 2024-08-19T11:20:51.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836529 1-42949672999' 2024-08-19T11:20:51.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:20:51.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T11:20:51.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509473 2024-08-19T11:20:51.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509473 2024-08-19T11:20:51.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836529 1-42949672999 2-64424509473' 2024-08-19T11:20:51.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:20:51.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836529 2024-08-19T11:20:51.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:20:51.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:20:51.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836529 2024-08-19T11:20:51.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:20:51.699 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836529 2024-08-19T11:20:51.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836529 2024-08-19T11:20:51.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-08-19T11:20:51.699 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:20:52.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836527 -lt 21474836529 2024-08-19T11:20:52.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:20:53.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:20:53.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:20:53.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836529 -lt 21474836529 2024-08-19T11:20:53.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:20:53.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672999 2024-08-19T11:20:53.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:20:53.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-19T11:20:53.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:20:53.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672999 2024-08-19T11:20:53.583 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672999 2024-08-19T11:20:53.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672999 2024-08-19T11:20:53.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2024-08-19T11:20:53.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T11:20:54.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672999 -lt 42949672999 2024-08-19T11:20:54.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:20:54.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509473 2024-08-19T11:20:54.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:20:54.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-19T11:20:54.051 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509473 2024-08-19T11:20:54.051 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:20:54.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509473 2024-08-19T11:20:54.053 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509473 2024-08-19T11:20:54.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-08-19T11:20:54.053 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T11:20:54.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509473 -lt 64424509473 2024-08-19T11:20:54.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:20:54.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2024-08-19T11:20:54.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:20:54.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:20:54.483 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:20:54.483 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:54.483 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:54.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-osd.0.asok 2024-08-19T11:20:54.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:20:54.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-osd.0.asok dump_osd_network 2024-08-19T11:20:54.591 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:20:54.592 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 1000, 2024-08-19T11:20:54.592 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [] 2024-08-19T11:20:54.592 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:20:54.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:54.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2024-08-19T11:20:54.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2024-08-19T11:20:54.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2024-08-19T11:20:54.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:54.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2024-08-19T11:20:54.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:20:54.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2024-08-19T11:20:54.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-08-19T11:20:54.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-08-19T11:20:54.682 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:20:54.682 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:54.682 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:54.682 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-mgr.x.asok 2024-08-19T11:20:54.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:20:54.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-mgr.x.asok dump_osd_network 2024-08-19T11:20:54.780 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:20:54.780 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 1000, 2024-08-19T11:20:54.780 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [] 2024-08-19T11:20:54.780 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:20:54.794 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:54.794 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2024-08-19T11:20:54.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2024-08-19T11:20:54.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:54.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2024-08-19T11:20:54.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2024-08-19T11:20:54.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:20:54.861 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2024-08-19T11:20:54.861 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:20:54.861 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:20:54.861 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:20:54.862 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:54.862 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:54.862 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-osd.0.asok 2024-08-19T11:20:54.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:20:54.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-osd.0.asok dump_osd_network 0 2024-08-19T11:20:54.973 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:20:54.973 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 0, 2024-08-19T11:20:54.973 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [ 2024-08-19T11:20:54.973 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:54.973 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:51 2024", 2024-08-19T11:20:54.973 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:54.973 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:54.974 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:20:54.974 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:54.974 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:54.974 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.569, 2024-08-19T11:20:54.974 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.607, 2024-08-19T11:20:54.974 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.588 2024-08-19T11:20:54.974 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.974 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:54.974 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.656, 2024-08-19T11:20:54.974 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.779, 2024-08-19T11:20:54.974 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.779 2024-08-19T11:20:54.974 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.974 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:54.975 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.656, 2024-08-19T11:20:54.975 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.779, 2024-08-19T11:20:54.975 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.779 2024-08-19T11:20:54.975 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.975 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.404 2024-08-19T11:20:54.975 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.975 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:54.975 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:51 2024", 2024-08-19T11:20:54.975 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:54.975 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:54.975 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:20:54.975 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:54.975 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:54.976 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.514, 2024-08-19T11:20:54.976 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.562, 2024-08-19T11:20:54.976 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.557 2024-08-19T11:20:54.976 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.976 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:54.976 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.593, 2024-08-19T11:20:54.976 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.706, 2024-08-19T11:20:54.976 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.706 2024-08-19T11:20:54.976 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.976 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:54.976 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.593, 2024-08-19T11:20:54.976 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.706, 2024-08-19T11:20:54.976 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.706 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.381 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:54 2024", 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.452, 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.524, 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.542 2024-08-19T11:20:54.977 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.542, 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.662, 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.695 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.542, 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.662, 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.695 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.356 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:54 2024", 2024-08-19T11:20:54.978 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.394, 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.500, 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.513 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.584, 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.658, 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.673 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.979 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:54.980 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.584, 2024-08-19T11:20:54.980 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.658, 2024-08-19T11:20:54.980 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.673 2024-08-19T11:20:54.980 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:54.980 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.328 2024-08-19T11:20:54.980 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-08-19T11:20:54.980 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-08-19T11:20:54.980 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:20:54.989 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:54.989 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2024-08-19T11:20:55.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2024-08-19T11:20:55.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:55.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2024-08-19T11:20:55.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2024-08-19T11:20:55.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:20:55.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2024-08-19T11:20:55.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-08-19T11:20:55.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-08-19T11:20:55.059 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:20:55.059 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:55.059 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:55.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-mgr.x.asok 2024-08-19T11:20:55.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:20:55.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-mgr.x.asok dump_osd_network 0 2024-08-19T11:20:55.154 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:20:55.155 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 0, 2024-08-19T11:20:55.155 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [ 2024-08-19T11:20:55.155 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:55.155 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:46 2024", 2024-08-19T11:20:55.155 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:55.155 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:55.155 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:20:55.155 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:55.155 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:55.155 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.612, 2024-08-19T11:20:55.155 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.616, 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.583 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.577, 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.514, 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.365 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.649, 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.779, 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.779 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.540 2024-08-19T11:20:55.156 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:49 2024", 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 2, 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.586, 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.586, 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.570 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.516, 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.464, 2024-08-19T11:20:55.157 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.303 2024-08-19T11:20:55.158 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.158 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:55.158 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.698, 2024-08-19T11:20:55.158 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.698, 2024-08-19T11:20:55.158 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.712 2024-08-19T11:20:55.158 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.158 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.316 2024-08-19T11:20:55.158 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.158 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:55.158 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:46 2024", 2024-08-19T11:20:55.158 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:55.158 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.557, 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.576, 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.555 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.487, 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.461, 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.312 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:55.159 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.609, 2024-08-19T11:20:55.160 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.706, 2024-08-19T11:20:55.160 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.706 2024-08-19T11:20:55.160 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.160 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.496 2024-08-19T11:20:55.160 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.160 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:55.160 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:50 2024", 2024-08-19T11:20:55.160 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:55.160 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 1, 2024-08-19T11:20:55.160 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:20:55.160 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:55.160 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:55.160 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.560, 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.516, 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.533 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.521, 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.426, 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.291 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.654, 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.654, 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.734 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.161 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.521 2024-08-19T11:20:55.162 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.162 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:55.162 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:50 2024", 2024-08-19T11:20:55.162 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:55.162 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 1, 2024-08-19T11:20:55.162 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 2, 2024-08-19T11:20:55.162 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:55.162 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:55.162 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.558, 2024-08-19T11:20:55.162 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.557, 2024-08-19T11:20:55.162 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.552 2024-08-19T11:20:55.162 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.162 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:55.163 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.515, 2024-08-19T11:20:55.163 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.508, 2024-08-19T11:20:55.163 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.267 2024-08-19T11:20:55.163 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.163 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:55.163 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.602, 2024-08-19T11:20:55.163 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.658, 2024-08-19T11:20:55.163 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.687 2024-08-19T11:20:55.163 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.163 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.550 2024-08-19T11:20:55.163 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.163 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:55.163 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:49 2024", 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 1, 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 0, 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.558, 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.546, 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.520 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.473, 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.444, 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.240 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.164 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:55.165 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.621, 2024-08-19T11:20:55.165 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.660, 2024-08-19T11:20:55.165 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.686 2024-08-19T11:20:55.165 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.165 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.635 2024-08-19T11:20:55.165 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.165 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:55.165 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:49 2024", 2024-08-19T11:20:55.165 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:55.165 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 2, 2024-08-19T11:20:55.165 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 0, 2024-08-19T11:20:55.165 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:55.165 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:55.166 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.526, 2024-08-19T11:20:55.166 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.554, 2024-08-19T11:20:55.166 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.535 2024-08-19T11:20:55.166 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.166 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:55.166 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.495, 2024-08-19T11:20:55.166 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.495, 2024-08-19T11:20:55.166 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.315 2024-08-19T11:20:55.166 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.166 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:55.166 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.553, 2024-08-19T11:20:55.166 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.663, 2024-08-19T11:20:55.166 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.680 2024-08-19T11:20:55.167 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.167 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.226 2024-08-19T11:20:55.167 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.167 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:55.167 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:49 2024", 2024-08-19T11:20:55.167 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:55.167 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:55.167 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:20:55.167 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "front", 2024-08-19T11:20:55.167 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:55.167 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.536, 2024-08-19T11:20:55.167 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.541, 2024-08-19T11:20:55.167 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.547 2024-08-19T11:20:55.168 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.168 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:55.168 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.454, 2024-08-19T11:20:55.168 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.440, 2024-08-19T11:20:55.168 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.320 2024-08-19T11:20:55.168 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.168 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:55.168 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.635, 2024-08-19T11:20:55.168 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.662, 2024-08-19T11:20:55.168 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.695 2024-08-19T11:20:55.168 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.168 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.455 2024-08-19T11:20:55.168 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.169 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:55.169 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:49 2024", 2024-08-19T11:20:55.169 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:55.169 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 2, 2024-08-19T11:20:55.169 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:20:55.169 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:55.169 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:55.169 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.527, 2024-08-19T11:20:55.169 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.530, 2024-08-19T11:20:55.169 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.536 2024-08-19T11:20:55.169 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.169 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:55.169 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.479, 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.442, 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.261 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.604, 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.621, 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.677 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.258 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:49 2024", 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:55.170 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 0, 2024-08-19T11:20:55.171 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 1, 2024-08-19T11:20:55.171 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:55.171 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:55.171 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.516, 2024-08-19T11:20:55.171 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.534, 2024-08-19T11:20:55.171 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.515 2024-08-19T11:20:55.171 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.171 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:55.171 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.446, 2024-08-19T11:20:55.171 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.425, 2024-08-19T11:20:55.171 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.301 2024-08-19T11:20:55.171 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.171 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:55.172 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.588, 2024-08-19T11:20:55.172 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.673, 2024-08-19T11:20:55.172 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.673 2024-08-19T11:20:55.172 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.172 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.584 2024-08-19T11:20:55.172 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.172 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:55.172 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:49 2024", 2024-08-19T11:20:55.172 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:55.172 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 1, 2024-08-19T11:20:55.172 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 0, 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.462, 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.516, 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.517 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.433, 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.433, 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.269 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.482, 2024-08-19T11:20:55.173 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.656, 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.684 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.439 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: "last update": "Mon Aug 19 11:20:49 2024", 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: "stale": false, 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: "from osd": 2, 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: "to osd": 0, 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: "interface": "back", 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: "average": { 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.469, 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.508, 2024-08-19T11:20:55.174 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.504 2024-08-19T11:20:55.175 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.175 INFO:tasks.workunit.client.0.smithi130.stdout: "min": { 2024-08-19T11:20:55.175 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.430, 2024-08-19T11:20:55.175 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.430, 2024-08-19T11:20:55.175 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.304 2024-08-19T11:20:55.175 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.175 INFO:tasks.workunit.client.0.smithi130.stdout: "max": { 2024-08-19T11:20:55.175 INFO:tasks.workunit.client.0.smithi130.stdout: "1min": 0.601, 2024-08-19T11:20:55.175 INFO:tasks.workunit.client.0.smithi130.stdout: "5min": 0.619, 2024-08-19T11:20:55.175 INFO:tasks.workunit.client.0.smithi130.stdout: "15min": 0.663 2024-08-19T11:20:55.175 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-08-19T11:20:55.176 INFO:tasks.workunit.client.0.smithi130.stdout: "last": 0.272 2024-08-19T11:20:55.176 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-08-19T11:20:55.176 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-08-19T11:20:55.176 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:20:55.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2024-08-19T11:20:55.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:55.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2024-08-19T11:20:55.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:55.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2024-08-19T11:20:55.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2024-08-19T11:20:55.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:20:55.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2024-08-19T11:20:55.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-08-19T11:20:55.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-08-19T11:20:55.239 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:20:55.239 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:55.239 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:55.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-mgr.x.asok 2024-08-19T11:20:55.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:20:55.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-mgr.x.asok dump_osd_network 99 2024-08-19T11:20:55.330 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:20:55.330 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 99, 2024-08-19T11:20:55.330 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [] 2024-08-19T11:20:55.330 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:20:55.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:55.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2024-08-19T11:20:55.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2024-08-19T11:20:55.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2024-08-19T11:20:55.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2024-08-19T11:20:55.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:20:55.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:20:55.378 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:20:55.378 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:55.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:55.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-osd.0.asok 2024-08-19T11:20:55.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2024-08-19T11:20:55.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35408/ceph-osd.0.asok dump_osd_network 98 2024-08-19T11:20:55.474 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-08-19T11:20:55.475 INFO:tasks.workunit.client.0.smithi130.stdout: "threshold": 98, 2024-08-19T11:20:55.475 INFO:tasks.workunit.client.0.smithi130.stdout: "entries": [] 2024-08-19T11:20:55.475 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-08-19T11:20:55.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2024-08-19T11:20:55.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2024-08-19T11:20:55.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2024-08-19T11:20:55.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2024-08-19T11:20:55.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-08-19T11:20:55.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-08-19T11:20:55.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:20:55.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-08-19T11:20:55.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:20:55.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:20:55.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:20:55.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:20:55.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:20:55.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:20:55.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:20:55.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:20:55.836 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:20:55.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:20:55.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:20:55.837 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:20:55.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:20:55.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:20:55.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:20:55.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:20:55.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:20:55.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:20:55.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-08-19T11:20:55.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:20:55.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:55.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:55.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.35408 2024-08-19T11:20:55.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:20:55.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:20:55.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-08-19T11:20:55.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-08-19T11:20:55.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-08-19T11:20:55.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-08-19T11:20:55.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-08-19T11:20:55.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:20:55.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-08-19T11:20:55.869 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:20:55.869 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:20:55.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:20:55.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:20:55.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:20:55.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:20:55.872 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:20:55.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:20:55.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:20:55.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:20:55.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:20:55.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:20:55.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:20:55.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:20:55.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:20:55.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:20:55.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:20:55.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:20:55.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-08-19T11:20:55.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:20:55.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:55.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:55.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.35408 2024-08-19T11:20:55.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:20:55.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:20:55.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-08-19T11:20:55.883 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:20:55.883 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:55.883 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:55.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.35408 2024-08-19T11:20:55.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:20:55.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:20:55.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:20:55.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-08-19T11:20:55.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2024-08-19T11:20:55.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2024-08-19T11:20:55.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2024-08-19T11:20:55.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2024-08-19T11:20:55.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2024-08-19T11:20:55.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2024-08-19T11:20:55.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-08-19T11:20:55.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:20:55.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:20:55.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:20:55.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-08-19T11:20:55.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-08-19T11:20:56.367 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:20:56.367 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:20:56.367 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:20:56.368 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:20:56.368 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:56.368 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:56.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:20:56.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:20:56.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:20:56.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:20:56.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:20:56.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:20:56.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:20:56.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:20:56.589 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:20:56.590 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:20:56.590 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:20:56.590 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:20:56.590 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:56.590 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:56.590 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-mon.a.asok 2024-08-19T11:20:56.590 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:20:56.590 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.35408/ceph-mon.a.asok config get fsid 2024-08-19T11:20:56.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:20:56.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:20:56.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:20:56.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:20:56.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:20:56.708 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:20:56.708 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:20:56.708 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:20:56.709 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:20:56.709 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:56.709 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:56.709 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35408/ceph-mon.a.asok 2024-08-19T11:20:56.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:20:56.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.35408/ceph-mon.a.asok config get mon_host 2024-08-19T11:20:56.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2024-08-19T11:20:56.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-08-19T11:20:56.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:20:56.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:20:56.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:20:56.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-08-19T11:20:56.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:20:57.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:20:57.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:20:57.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:20:57.752 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:20:57.752 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:57.753 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:57.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:20:57.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:20:57.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:20:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2024-08-19T11:20:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-08-19T11:20:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:20:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:20:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:20:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-08-19T11:20:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-08-19T11:20:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:20:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:20:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:20:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-08-19T11:20:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-08-19T11:20:57.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:20:57.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-08-19T11:20:57.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-08-19T11:20:57.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:20:57.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:20:57.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:20:57.790 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:20:57.790 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:20:57.790 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:20:57.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:20:57.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:20:57.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:20:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:20:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:20:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-08-19T11:20:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-08-19T11:20:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:20:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:20:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:20:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:20:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:20:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:20:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-08-19T11:20:57.792 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:20:57.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ee4c6cbd-37ca-4138-ac50-ecac435694a9 2024-08-19T11:20:57.792 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 ee4c6cbd-37ca-4138-ac50-ecac435694a9 2024-08-19T11:20:57.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ee4c6cbd-37ca-4138-ac50-ecac435694a9' 2024-08-19T11:20:57.792 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:20:57.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZKsNmrj5TMBAARfXK4b5ZamoRmE5cdN+u6g== 2024-08-19T11:20:57.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZKsNmrj5TMBAARfXK4b5ZamoRmE5cdN+u6g=="}' 2024-08-19T11:20:57.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ee4c6cbd-37ca-4138-ac50-ecac435694a9 -i td/network-ping/0/new.json 2024-08-19T11:20:58.130 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:20:58.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-08-19T11:20:58.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCZKsNmrj5TMBAARfXK4b5ZamoRmE5cdN+u6g== --osd-uuid ee4c6cbd-37ca-4138-ac50-ecac435694a9 2024-08-19T11:20:58.180 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:20:58.181+0000 7fdfe4eaf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:20:58.199 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:20:58.197+0000 7fdfe4eaf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:20:58.200 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:20:58.201+0000 7fdfe4eaf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:20:58.200 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:20:58.201+0000 7fdfe4eaf7c0 -1 bdev(0x557af88ebc00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:20:58.200 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:20:58.201+0000 7fdfe4eaf7c0 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-08-19T11:21:04.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-08-19T11:21:04.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:21:04.819 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:21:04.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:21:04.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:21:05.376 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:21:05.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:21:05.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:21:05.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:21:05.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:21:05.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:21:05.409 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:05.409+0000 7f0dc9b767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:21:05.428 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:05.429+0000 7f0dc9b767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:21:05.430 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:05.429+0000 7f0dc9b767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:21:05.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:21:05.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:21:05.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:21:05.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:21:05.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:21:05.794 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:21:05.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:05.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:21:05.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:05.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:06.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:06.711 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:06.709+0000 7f0dc9b767c0 -1 Falling back to public interface 2024-08-19T11:21:07.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:07.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:07.247 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:21:07.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:21:07.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:07.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:07.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:08.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:08.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:08.691 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:21:08.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:21:08.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:08.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:09.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:09.663 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:09.660+0000 7f0dc9b767c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:21:10.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:10.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:10.140 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:21:10.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:21:10.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:10.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:10.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:11.606 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:21:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:21:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:12.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:13.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:13.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:13.078 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:21:13.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:21:13.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:13.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:13.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:14.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:14.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:14.622 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:21:14.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:21:14.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:14.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:15.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:16.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:16.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:16.063 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:21:16.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:21:16.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:16.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:16.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:17.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:17.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:17.619 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:21:17.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:21:17.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:17.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:18.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:19.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:19.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:19.073 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:21:19.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:21:19.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:19.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:19.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:20.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:20.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:20.671 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:21:20.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:21:20.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:20.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:21.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:22.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:22.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:22.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:21:22.134 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:21:22.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:22.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:22.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:23.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:23.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:23.743 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:21:23.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:21:23.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:23.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:24.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:25.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:25.891 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:21:25.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:25.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:21:25.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:25.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:26.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:27.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:27.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:27.530 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:21:27.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:21:27.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:27.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:28.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:29.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:29.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:29.267 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:21:29.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:21:29.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:29.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:30.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:31.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:31.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:31.071 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:21:31.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:21:31.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:31.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:32.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:33.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:33.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:33.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:21:33.203 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:21:33.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:33.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:34.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:35.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:35.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:35.245 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:21:35.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:21:35.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:35.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:21:35.716 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3975699934,v1:127.0.0.1:6803/3975699934] [v2:127.0.0.1:6804/3975699934,v1:127.0.0.1:6805/3975699934] exists,up ee4c6cbd-37ca-4138-ac50-ecac435694a9 2024-08-19T11:21:35.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:21:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:21:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:21:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2024-08-19T11:21:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-08-19T11:21:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:21:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T11:21:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:21:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-08-19T11:21:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-08-19T11:21:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:21:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:21:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:21:35.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:21:35.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:21:35.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:21:35.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-08-19T11:21:35.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-08-19T11:21:35.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:21:35.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:21:35.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:21:35.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:21:35.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:21:35.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:21:35.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-08-19T11:21:35.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:21:35.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e2b5b8ef-c2b1-4e83-b130-0a9c6e9f07e9 2024-08-19T11:21:35.724 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 e2b5b8ef-c2b1-4e83-b130-0a9c6e9f07e9 2024-08-19T11:21:35.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e2b5b8ef-c2b1-4e83-b130-0a9c6e9f07e9' 2024-08-19T11:21:35.724 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:21:35.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/KsNmnhMwLBAAOPaqa1YALBq6Qid+UFoMFA== 2024-08-19T11:21:35.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/KsNmnhMwLBAAOPaqa1YALBq6Qid+UFoMFA=="}' 2024-08-19T11:21:35.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e2b5b8ef-c2b1-4e83-b130-0a9c6e9f07e9 -i td/network-ping/1/new.json 2024-08-19T11:21:36.378 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:21:36.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-08-19T11:21:36.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/KsNmnhMwLBAAOPaqa1YALBq6Qid+UFoMFA== --osd-uuid e2b5b8ef-c2b1-4e83-b130-0a9c6e9f07e9 2024-08-19T11:21:36.426 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:36.424+0000 7f86e75787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:21:36.450 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:36.448+0000 7f86e75787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:21:36.451 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:36.448+0000 7f86e75787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:21:36.451 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:36.452+0000 7f86e75787c0 -1 bdev(0x55f6fbe3bc00 td/network-ping/1/block) open stat got: (1) Operation not permitted 2024-08-19T11:21:36.452 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:36.452+0000 7f86e75787c0 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-08-19T11:21:43.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-08-19T11:21:43.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:21:43.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T11:21:43.642 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-08-19T11:21:43.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:21:44.214 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-08-19T11:21:44.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T11:21:44.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:21:44.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:21:44.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:21:44.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:21:44.248 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:44.248+0000 7faf1383f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:21:44.250 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:44.248+0000 7faf1383f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:21:44.252 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:44.252+0000 7faf1383f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:21:44.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T11:21:44.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:21:44.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T11:21:44.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:21:44.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:21:44.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:44.693 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:21:44.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:21:44.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:44.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:21:45.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:45.518 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:45.516+0000 7faf1383f7c0 -1 Falling back to public interface 2024-08-19T11:21:46.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:46.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:46.162 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:21:46.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:21:46.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:46.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:21:46.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:47.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:47.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:47.613 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:21:47.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:21:47.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:47.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:21:48.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:48.474 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:21:48.472+0000 7faf1383f7c0 -1 osd.1 0 log_to_monitors true 2024-08-19T11:21:49.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:49.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:49.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:21:49.032 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:21:49.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:49.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:21:49.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:50.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:50.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:50.721 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:21:50.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:21:50.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:50.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:21:51.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:52.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:52.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:52.336 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:21:52.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:21:52.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:52.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:21:52.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:53.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:53.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:53.773 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:21:53.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:21:53.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:53.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:21:54.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:55.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:55.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:55.421 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:21:55.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:21:55.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:55.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:21:55.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:56.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:56.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:56.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:21:56.866 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:21:56.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:56.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:21:57.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:21:58.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:21:58.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:21:58.599 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:21:58.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:21:58.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:21:58.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:21:59.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:00.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:00.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:00.117 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:22:00.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:22:00.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:00.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:22:00.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:01.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:01.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:01.587 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:22:01.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:22:01.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:01.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:22:02.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:03.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:03.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:03.052 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:22:03.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:22:03.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:03.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:22:04.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:05.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:05.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:05.044 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:22:05.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:22:05.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:05.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:22:05.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:06.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:06.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:06.989 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:22:06.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:22:06.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:06.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:22:08.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:09.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:09.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:09.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:22:09.231 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:22:09.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:09.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:22:10.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:11.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:11.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:11.329 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:22:11.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:22:11.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:11.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:22:12.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:13.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:13.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:13.141 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:22:13.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:22:13.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:13.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:22:14.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:15.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:15.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:15.005 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:22:15.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:22:15.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:15.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:22:15.464 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4206488753,v1:127.0.0.1:6811/4206488753] [v2:127.0.0.1:6812/4206488753,v1:127.0.0.1:6813/4206488753] exists,up e2b5b8ef-c2b1-4e83-b130-0a9c6e9f07e9 2024-08-19T11:22:15.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:22:15.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:22:15.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:22:15.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2024-08-19T11:22:15.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-08-19T11:22:15.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:22:15.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T11:22:15.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:22:15.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-08-19T11:22:15.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-08-19T11:22:15.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:22:15.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:22:15.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' 2024-08-19T11:22:15.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:22:15.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:22:15.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:22:15.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-08-19T11:22:15.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-08-19T11:22:15.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:22:15.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:22:15.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:22:15.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:22:15.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:22:15.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:22:15.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-08-19T11:22:15.471 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:22:15.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=79093767-f365-474f-8fb8-c3f4dbbd1a3c 2024-08-19T11:22:15.472 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 79093767-f365-474f-8fb8-c3f4dbbd1a3c 2024-08-19T11:22:15.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 79093767-f365-474f-8fb8-c3f4dbbd1a3c' 2024-08-19T11:22:15.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:22:15.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDnKsNmRoWRHRAAGZSPsWPHe10ySXInJYIC5g== 2024-08-19T11:22:15.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDnKsNmRoWRHRAAGZSPsWPHe10ySXInJYIC5g=="}' 2024-08-19T11:22:15.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 79093767-f365-474f-8fb8-c3f4dbbd1a3c -i td/network-ping/2/new.json 2024-08-19T11:22:16.003 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:22:16.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-08-19T11:22:16.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnKsNmRoWRHRAAGZSPsWPHe10ySXInJYIC5g== --osd-uuid 79093767-f365-474f-8fb8-c3f4dbbd1a3c 2024-08-19T11:22:16.056 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:22:16.056+0000 7f7f1839d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:22:16.087 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:22:16.084+0000 7f7f1839d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:22:16.088 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:22:16.088+0000 7f7f1839d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:22:16.089 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:22:16.088+0000 7f7f1839d7c0 -1 bdev(0x55d68cd32800 td/network-ping/2/block) open stat got: (1) Operation not permitted 2024-08-19T11:22:16.089 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:22:16.088+0000 7f7f1839d7c0 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-08-19T11:22:22.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-08-19T11:22:22.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:22:22.844 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-08-19T11:22:22.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T11:22:22.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:22:23.426 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-08-19T11:22:23.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T11:22:23.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2e9ee693-8057-46ab-9c67-ad008e28537b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35408/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:22:23.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:22:23.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:22:23.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:22:23.459 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:22:23.456+0000 7f50983487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:22:23.492 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:22:23.492+0000 7f50983487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:22:23.493 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:22:23.492+0000 7f50983487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:22:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T11:22:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:22:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T11:22:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:22:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:22:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:23.939 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:22:23.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:22:23.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:23.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:24.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:24.742 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:22:24.740+0000 7f50983487c0 -1 Falling back to public interface 2024-08-19T11:22:25.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:25.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:25.386 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:22:25.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:22:25.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:25.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:25.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:26.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:26.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:26.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:22:26.826 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:22:26.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:26.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:27.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:27.743 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:22:27.744+0000 7f50983487c0 -1 osd.2 0 log_to_monitors true 2024-08-19T11:22:28.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:28.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:28.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:22:28.258 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:22:28.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:28.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:28.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:29.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:29.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:29.740 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:22:29.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:22:29.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:29.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:30.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:31.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:31.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:31.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:22:31.207 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:22:31.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:31.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:31.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:32.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:32.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:32.701 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:22:32.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:22:32.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:32.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:33.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:34.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:34.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:34.227 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:22:34.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:22:34.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:34.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:34.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:35.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:35.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:35.784 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:22:35.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:22:35.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:35.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:36.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:37.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:37.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:37.248 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:22:37.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:22:37.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:37.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:37.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:38.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:38.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:38.823 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:22:38.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:22:38.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:38.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:39.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:40.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:40.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:40.431 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:22:40.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:22:40.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:40.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:41.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:42.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:42.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:42.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:22:42.176 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:22:42.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:42.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:42.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:43.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:43.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:43.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:22:43.769 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:22:43.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:43.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:44.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:45.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:45.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:45.279 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:22:45.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:22:45.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:45.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:46.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:47.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:47.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:47.481 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:22:47.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:22:47.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:47.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:48.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:49.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:49.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:49.319 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:22:49.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:22:49.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:49.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:50.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:51.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:51.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:51.175 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:22:51.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:22:51.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:51.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:52.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:53.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:53.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:53.093 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:22:53.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:22:53.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:53.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:54.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:22:55.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:22:55.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:22:55.021 INFO:tasks.workunit.client.0.smithi130.stdout:19 2024-08-19T11:22:55.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T11:22:55.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:22:55.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:22:55.461 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4581375,v1:127.0.0.1:6819/4581375] [v2:127.0.0.1:6820/4581375,v1:127.0.0.1:6821/4581375] exists,up 79093767-f365-474f-8fb8-c3f4dbbd1a3c 2024-08-19T11:22:55.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:22:55.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:22:55.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:22:55.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2024-08-19T11:23:00.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2024-08-19T11:23:01.026 INFO:tasks.workunit.client.0.smithi130.stderr:bucket 'dc1' already exists 2024-08-19T11:23:01.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2024-08-19T11:23:01.578 INFO:tasks.workunit.client.0.smithi130.stderr:bucket 'dc2' already exists 2024-08-19T11:23:01.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2024-08-19T11:23:02.263 INFO:tasks.workunit.client.0.smithi130.stderr:bucket 'dc3' already exists 2024-08-19T11:23:02.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2024-08-19T11:23:02.899 INFO:tasks.workunit.client.0.smithi130.stderr:bucket 'rack1' already exists 2024-08-19T11:23:02.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2024-08-19T11:23:03.562 INFO:tasks.workunit.client.0.smithi130.stderr:bucket 'rack2' already exists 2024-08-19T11:23:03.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2024-08-19T11:23:04.240 INFO:tasks.workunit.client.0.smithi130.stderr:bucket 'rack3' already exists 2024-08-19T11:23:04.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2024-08-19T11:23:05.030 INFO:tasks.workunit.client.0.smithi130.stderr:bucket 'host1' already exists 2024-08-19T11:23:05.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2024-08-19T11:23:05.653 INFO:tasks.workunit.client.0.smithi130.stderr:bucket 'host2' already exists 2024-08-19T11:23:05.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2024-08-19T11:23:06.199 INFO:tasks.workunit.client.0.smithi130.stderr:bucket 'host3' already exists 2024-08-19T11:23:06.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2024-08-19T11:23:06.937 INFO:tasks.workunit.client.0.smithi130.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2024-08-19T11:23:06.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2024-08-19T11:23:07.501 INFO:tasks.workunit.client.0.smithi130.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2024-08-19T11:23:07.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2024-08-19T11:23:08.113 INFO:tasks.workunit.client.0.smithi130.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2024-08-19T11:23:08.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2024-08-19T11:23:08.740 INFO:tasks.workunit.client.0.smithi130.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2024-08-19T11:23:08.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2024-08-19T11:23:09.375 INFO:tasks.workunit.client.0.smithi130.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2024-08-19T11:23:09.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2024-08-19T11:23:10.004 INFO:tasks.workunit.client.0.smithi130.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2024-08-19T11:23:10.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2024-08-19T11:23:10.585 INFO:tasks.workunit.client.0.smithi130.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2024-08-19T11:23:10.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2024-08-19T11:23:11.178 INFO:tasks.workunit.client.0.smithi130.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2024-08-19T11:23:11.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2024-08-19T11:23:11.886 INFO:tasks.workunit.client.0.smithi130.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2024-08-19T11:23:11.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2024-08-19T11:23:12.398 INFO:tasks.workunit.client.0.smithi130.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2024-08-19T11:23:12.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2024-08-19T11:23:13.099 INFO:tasks.workunit.client.0.smithi130.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2024-08-19T11:23:13.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2024-08-19T11:23:13.717 INFO:tasks.workunit.client.0.smithi130.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2024-08-19T11:23:13.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2024-08-19T11:23:14.404 INFO:tasks.workunit.client.0.smithi130.stderr:rule myrule already exists 2024-08-19T11:23:14.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2024-08-19T11:23:14.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 16 replicated myrule 2024-08-19T11:23:15.069 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'foo' already exists 2024-08-19T11:23:15.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:23:16.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-08-19T11:23:16.276 INFO:tasks.workunit.client.0.smithi130.stdout:hints = 1 2024-08-19T11:23:16.276 INFO:tasks.workunit.client.0.smithi130.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-08-19T11:23:16.277 INFO:tasks.workunit.client.0.smithi130.stdout:Object prefix: benchmark_data_smithi130_47790 2024-08-19T11:23:16.277 INFO:tasks.workunit.client.0.smithi130.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-19T11:23:16.277 INFO:tasks.workunit.client.0.smithi130.stdout: 0 0 0 0 0 0 - 0 2024-08-19T11:23:17.276 INFO:tasks.workunit.client.0.smithi130.stdout: 1 16 35 19 0.0742095 0.0742188 0.293037 0.677154 2024-08-19T11:23:18.276 INFO:tasks.workunit.client.0.smithi130.stdout: 2 16 94 78 0.152321 0.230469 0.283309 0.392219 2024-08-19T11:23:19.277 INFO:tasks.workunit.client.0.smithi130.stdout: 3 16 147 131 0.170546 0.207031 0.31102 0.351538 2024-08-19T11:23:20.277 INFO:tasks.workunit.client.0.smithi130.stdout: 4 16 206 190 0.185518 0.230469 0.292759 0.327341 2024-08-19T11:23:21.277 INFO:tasks.workunit.client.0.smithi130.stdout: 5 16 259 243 0.189815 0.207031 0.285137 0.317227 2024-08-19T11:23:22.277 INFO:tasks.workunit.client.0.smithi130.stdout: 6 16 318 302 0.196585 0.230469 0.303552 0.309739 2024-08-19T11:23:23.277 INFO:tasks.workunit.client.0.smithi130.stdout: 7 16 371 355 0.198073 0.207031 0.310332 0.308056 2024-08-19T11:23:24.277 INFO:tasks.workunit.client.0.smithi130.stdout: 8 16 430 414 0.202118 0.230469 0.277507 0.304409 2024-08-19T11:23:25.277 INFO:tasks.workunit.client.0.smithi130.stdout: 9 16 483 467 0.202661 0.207031 0.278513 0.301267 2024-08-19T11:23:26.278 INFO:tasks.workunit.client.0.smithi130.stdout: 10 16 548 532 0.207781 0.253906 0.239631 0.295102 2024-08-19T11:23:27.278 INFO:tasks.workunit.client.0.smithi130.stdout:Total time run: 10.1219 2024-08-19T11:23:27.278 INFO:tasks.workunit.client.0.smithi130.stdout:Total writes made: 548 2024-08-19T11:23:27.278 INFO:tasks.workunit.client.0.smithi130.stdout:Write size: 4096 2024-08-19T11:23:27.278 INFO:tasks.workunit.client.0.smithi130.stdout:Object size: 4096 2024-08-19T11:23:27.278 INFO:tasks.workunit.client.0.smithi130.stdout:Bandwidth (MB/sec): 0.211485 2024-08-19T11:23:27.278 INFO:tasks.workunit.client.0.smithi130.stdout:Stddev Bandwidth: 0.0494723 2024-08-19T11:23:27.279 INFO:tasks.workunit.client.0.smithi130.stdout:Max bandwidth (MB/sec): 0.253906 2024-08-19T11:23:27.279 INFO:tasks.workunit.client.0.smithi130.stdout:Min bandwidth (MB/sec): 0.0742188 2024-08-19T11:23:27.279 INFO:tasks.workunit.client.0.smithi130.stdout:Average IOPS: 54 2024-08-19T11:23:27.279 INFO:tasks.workunit.client.0.smithi130.stdout:Stddev IOPS: 12.6649 2024-08-19T11:23:27.279 INFO:tasks.workunit.client.0.smithi130.stdout:Max IOPS: 65 2024-08-19T11:23:27.279 INFO:tasks.workunit.client.0.smithi130.stdout:Min IOPS: 19 2024-08-19T11:23:27.279 INFO:tasks.workunit.client.0.smithi130.stdout:Average Latency(s): 0.293248 2024-08-19T11:23:27.279 INFO:tasks.workunit.client.0.smithi130.stdout:Stddev Latency(s): 0.0851048 2024-08-19T11:23:27.279 INFO:tasks.workunit.client.0.smithi130.stdout:Max latency(s): 0.804807 2024-08-19T11:23:27.279 INFO:tasks.workunit.client.0.smithi130.stdout:Min latency(s): 0.172857 2024-08-19T11:23:27.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2024-08-19T11:23:37.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2024-08-19T11:23:37.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:23:37.359 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:23:37.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-19T11:23:37.797 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-08-19T11:23:37.797 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-08-19T11:23:37.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:23:37.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:23:37.797 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:23:38.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836515 2024-08-19T11:23:38.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836515 2024-08-19T11:23:38.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836515' 2024-08-19T11:23:38.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:23:38.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T11:23:38.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672982 2024-08-19T11:23:38.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672982 2024-08-19T11:23:38.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836515 1-42949672982' 2024-08-19T11:23:38.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:23:38.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T11:23:38.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509456 2024-08-19T11:23:38.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509456 2024-08-19T11:23:38.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836515 1-42949672982 2-64424509456' 2024-08-19T11:23:38.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:23:38.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836515 2024-08-19T11:23:38.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:23:38.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:23:38.322 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836515 2024-08-19T11:23:38.322 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:23:38.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836515 2024-08-19T11:23:38.323 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836515 2024-08-19T11:23:38.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-08-19T11:23:38.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:23:38.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836515 -lt 21474836515 2024-08-19T11:23:38.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:23:38.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672982 2024-08-19T11:23:38.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:23:38.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-19T11:23:38.778 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672982 2024-08-19T11:23:38.778 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:23:38.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672982 2024-08-19T11:23:38.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-08-19T11:23:38.780 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672982 2024-08-19T11:23:38.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T11:23:39.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672982 -lt 42949672982 2024-08-19T11:23:39.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:23:39.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509456 2024-08-19T11:23:39.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:23:39.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-19T11:23:39.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509456 2024-08-19T11:23:39.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:23:39.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509456 2024-08-19T11:23:39.245 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509456 2024-08-19T11:23:39.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-08-19T11:23:39.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T11:23:39.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509456 -lt 64424509456 2024-08-19T11:23:39.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2024-08-19T11:23:39.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2024-08-19T11:23:40.171 INFO:tasks.workunit.client.0.smithi130.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.598ms); Slow OSD heartbeats on front (longest 0.607ms); 1 pool(s) do not have an application enabled 2024-08-19T11:23:40.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2024-08-19T11:23:40.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2024-08-19T11:23:40.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2024-08-19T11:23:40.751 INFO:tasks.workunit.client.0.smithi130.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.598ms); Slow OSD heartbeats on front (longest 0.607ms); 1 pool(s) do not have an application enabled 2024-08-19T11:23:40.751 INFO:tasks.workunit.client.0.smithi130.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.598ms) 2024-08-19T11:23:40.751 INFO:tasks.workunit.client.0.smithi130.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.598 msec possibly improving 2024-08-19T11:23:40.751 INFO:tasks.workunit.client.0.smithi130.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.597 msec 2024-08-19T11:23:40.751 INFO:tasks.workunit.client.0.smithi130.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.568 msec 2024-08-19T11:23:40.751 INFO:tasks.workunit.client.0.smithi130.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.509 msec 2024-08-19T11:23:40.751 INFO:tasks.workunit.client.0.smithi130.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.507 msec 2024-08-19T11:23:40.751 INFO:tasks.workunit.client.0.smithi130.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.503 msec 2024-08-19T11:23:40.752 INFO:tasks.workunit.client.0.smithi130.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.607ms) 2024-08-19T11:23:40.752 INFO:tasks.workunit.client.0.smithi130.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.607 msec 2024-08-19T11:23:40.752 INFO:tasks.workunit.client.0.smithi130.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.593 msec 2024-08-19T11:23:40.752 INFO:tasks.workunit.client.0.smithi130.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.592 msec 2024-08-19T11:23:40.752 INFO:tasks.workunit.client.0.smithi130.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.566 msec 2024-08-19T11:23:40.752 INFO:tasks.workunit.client.0.smithi130.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.525 msec 2024-08-19T11:23:40.752 INFO:tasks.workunit.client.0.smithi130.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.514 msec 2024-08-19T11:23:40.752 INFO:tasks.workunit.client.0.smithi130.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-08-19T11:23:40.752 INFO:tasks.workunit.client.0.smithi130.stdout: application not enabled on pool 'foo' 2024-08-19T11:23:40.752 INFO:tasks.workunit.client.0.smithi130.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-08-19T11:23:40.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2024-08-19T11:23:40.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2024-08-19T11:23:40.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2024-08-19T11:23:40.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2024-08-19T11:23:40.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-08-19T11:23:40.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-08-19T11:23:40.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:23:40.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-08-19T11:23:40.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:23:40.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:23:40.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:23:40.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:23:40.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:23:40.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:23:40.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:23:40.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:23:40.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:23:40.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:23:40.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:23:40.883 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:23:40.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:23:40.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:23:40.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:23:40.885 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:23:40.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:23:40.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:23:40.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-08-19T11:23:40.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:23:40.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:23:40.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:23:40.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.35408 2024-08-19T11:23:40.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:23:40.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:23:40.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2331: main: code=0 2024-08-19T11:23:40.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: teardown td/network-ping 0 2024-08-19T11:23:40.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-08-19T11:23:40.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-19T11:23:40.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-08-19T11:23:40.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:23:40.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:23:40.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:23:40.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:23:40.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:23:40.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:23:40.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:23:40.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:23:40.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:23:40.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:23:40.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:23:40.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:23:40.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:23:40.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:23:40.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:23:40.912 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:23:40.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:23:40.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-19T11:23:40.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-08-19T11:23:40.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:23:40.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:23:40.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35408 2024-08-19T11:23:40.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.35408 2024-08-19T11:23:40.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:23:40.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:23:40.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: main: return 0 2024-08-19T11:23:40.917 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-19T11:23:40.917 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-19T11:23:40.926 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2024-08-19T11:23:40.926 DEBUG:teuthology.orchestra.run.smithi130:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2024-08-19T11:23:40.980 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-19T11:23:40.986 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-19T11:23:40.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-19T11:23:40.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-19T11:23:40.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export PYTHONWARNINGS=ignore 2024-08-19T11:23:40.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: PYTHONWARNINGS=ignore 2024-08-19T11:23:40.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: export CEPH_CONF=/dev/null 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: CEPH_CONF=/dev/null 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: unset CEPH_ARGS 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: local code 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: run td/ok-to-stop 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2024-08-19T11:23:40.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2024-08-19T11:23:40.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2024-08-19T11:23:40.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2024-08-19T11:23:40.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2024-08-19T11:23:40.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2024-08-19T11:23:40.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none ' 2024-08-19T11:23:40.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none ' 2024-08-19T11:23:40.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none ' 2024-08-19T11:23:40.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2024-08-19T11:23:40.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-19T11:23:40.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2024-08-19T11:23:40.990 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_0_osd 2024-08-19T11:23:40.990 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_1_mon_checks 2024-08-19T11:23:40.991 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_2_mons_checks 2024-08-19T11:23:40.991 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_3_mons_checks 2024-08-19T11:23:40.991 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_4_mons_checks 2024-08-19T11:23:40.991 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_5_mons_checks' 2024-08-19T11:23:40.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-08-19T11:23:40.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-08-19T11:23:40.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-08-19T11:23:40.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-08-19T11:23:40.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:23:40.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:23:40.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:23:40.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:23:40.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:23:40.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:23:40.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:23:40.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:23:40.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:23:40.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:23:40.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:23:40.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:23:40.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:23:40.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:23:40.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:23:40.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:23:40.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:23:40.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:23:40.997 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:23:40.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:23:40.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:23:40.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:23:41.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:23:41.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:23:41.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:23:41.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:23:41.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:23:41.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:23:41.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-08-19T11:23:41.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:23:41.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:23:41.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:23:41.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.48100 2024-08-19T11:23:41.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:23:41.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-19T11:23:41.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-19T11:23:41.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:23:41.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-08-19T11:23:41.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2024-08-19T11:23:41.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2024-08-19T11:23:41.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-08-19T11:23:41.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-08-19T11:23:41.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:23:41.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:23:41.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:23:41.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:23:41.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-08-19T11:23:41.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-08-19T11:23:41.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:23:41.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:23:41.523 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:23:41.523 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:23:41.523 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:23:41.523 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:23:41.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:23:41.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-08-19T11:23:41.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:23:41.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:23:41.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:23:41.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:23:41.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:23:41.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:23:41.736 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:23:41.736 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:23:41.736 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:23:41.737 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:23:41.737 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:23:41.737 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:23:41.737 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:23:41.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:23:41.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get fsid 2024-08-19T11:23:41.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:23:41.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:23:41.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:23:41.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:23:41.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:23:41.849 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:23:41.849 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:23:41.849 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:23:41.850 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:23:41.850 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:23:41.850 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:23:41.850 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:23:41.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:23:41.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get mon_host 2024-08-19T11:23:41.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2024-08-19T11:23:41.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-08-19T11:23:41.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:23:41.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:23:41.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:23:41.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-08-19T11:23:41.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:23:42.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:23:42.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:23:42.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:23:42.865 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:23:42.865 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:23:42.865 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:23:42.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:23:42.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:23:42.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:23:42.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2024-08-19T11:23:42.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-08-19T11:23:42.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:23:42.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:23:42.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:23:42.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-08-19T11:23:42.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-08-19T11:23:42.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:23:42.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:23:42.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:23:42.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-08-19T11:23:42.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-08-19T11:23:42.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:23:42.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:23:42.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-08-19T11:23:42.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:23:42.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:23:42.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:23:42.911 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:23:42.911 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:23:42.911 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:23:42.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:23:42.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:23:42.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:23:42.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:23:42.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:23:42.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-08-19T11:23:42.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-08-19T11:23:42.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:23:42.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:23:42.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:23:42.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:23:42.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:23:42.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:23:42.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-08-19T11:23:42.914 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:23:42.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2efb8a1-46cc-4149-b0d5-f78e0f006f36 2024-08-19T11:23:42.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b2efb8a1-46cc-4149-b0d5-f78e0f006f36' 2024-08-19T11:23:42.915 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 b2efb8a1-46cc-4149-b0d5-f78e0f006f36 2024-08-19T11:23:42.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:23:42.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA+K8Nm81IqNxAARRHUxYSFq0wb29SNZF7HzQ== 2024-08-19T11:23:42.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA+K8Nm81IqNxAARRHUxYSFq0wb29SNZF7HzQ=="}' 2024-08-19T11:23:42.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2efb8a1-46cc-4149-b0d5-f78e0f006f36 -i td/ok-to-stop/0/new.json 2024-08-19T11:23:43.229 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:23:43.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-08-19T11:23:43.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA+K8Nm81IqNxAARRHUxYSFq0wb29SNZF7HzQ== --osd-uuid b2efb8a1-46cc-4149-b0d5-f78e0f006f36 2024-08-19T11:23:43.282 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:23:43.279+0000 7f248044d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:23:43.306 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:23:43.303+0000 7f248044d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:23:43.307 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:23:43.307+0000 7f248044d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:23:43.308 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:23:43.307+0000 7f248044d7c0 -1 bdev(0x5600a3c61c00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:23:43.308 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:23:43.307+0000 7f248044d7c0 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-08-19T11:23:50.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-08-19T11:23:50.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:23:50.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:23:50.218 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:23:50.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:23:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:23:50.767 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:23:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:23:50.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:23:50.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:23:50.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:23:50.796 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:23:50.795+0000 7f67cad747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:23:50.828 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:23:50.827+0000 7f67cad747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:23:50.830 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:23:50.827+0000 7f67cad747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:23:51.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:23:51.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:23:51.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:23:51.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:23:51.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:23:51.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:23:51.236 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:23:51.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:23:51.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:23:51.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:23:51.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:23:51.849 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:23:51.847+0000 7f67cad747c0 -1 Falling back to public interface 2024-08-19T11:23:52.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:23:52.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:23:52.664 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:23:52.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:23:52.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:23:52.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:23:53.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:23:54.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:23:54.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:23:54.082 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:23:54.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:23:54.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:23:54.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:23:54.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:23:54.673 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:23:54.671+0000 7f67cad747c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:23:55.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:23:55.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:23:55.518 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:23:55.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:23:55.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:23:55.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:23:55.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:23:56.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:23:56.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:23:56.976 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:23:56.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:23:56.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:23:56.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:23:57.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:23:58.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:23:58.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:23:58.448 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:23:58.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:23:58.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:23:58.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:23:58.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:23:59.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:23:59.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:23:59.952 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:23:59.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:23:59.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:23:59.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:00.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:01.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:01.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:01.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:24:01.423 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:24:01.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:01.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:01.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:02.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:02.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:02.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:24:02.998 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:24:02.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:02.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:03.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:04.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:04.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:04.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:24:04.451 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:24:04.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:04.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:04.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:05.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:05.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:05.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:24:05.974 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:24:05.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:05.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:06.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:07.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:07.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:07.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:24:07.448 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:24:07.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:07.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:08.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:09.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:09.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:09.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:24:09.082 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:24:09.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:09.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:09.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:10.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:10.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:10.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:24:10.618 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:24:10.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:10.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:11.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:12.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:12.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:12.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:24:12.473 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:24:12.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:12.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:13.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:14.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:14.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:14.274 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:24:14.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:24:14.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:14.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:15.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:16.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:16.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:16.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:24:16.596 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:24:16.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:16.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:17.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:18.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:18.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:18.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:24:18.560 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:24:18.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:18.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:19.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:20.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:20.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:20.437 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:24:20.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:24:20.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:20.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:20.908 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3156059182,v1:127.0.0.1:6803/3156059182] [v2:127.0.0.1:6804/3156059182,v1:127.0.0.1:6805/3156059182] exists,up b2efb8a1-46cc-4149-b0d5-f78e0f006f36 2024-08-19T11:24:20.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:24:20.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:24:20.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:24:20.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2024-08-19T11:24:20.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-08-19T11:24:20.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-08-19T11:24:20.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=a 2024-08-19T11:24:20.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-08-19T11:24:20.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/a 2024-08-19T11:24:20.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-08-19T11:24:20.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:24:20.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:24:20.911 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:24:20.911 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:24:20.911 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:24:20.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:24:20.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-08-19T11:24:20.942 INFO:tasks.workunit.client.0.smithi130.stdout:starting mds.a at 2024-08-19T11:24:20.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2024-08-19T11:24:21.529 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'meta' already exists 2024-08-19T11:24:21.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2024-08-19T11:24:22.449 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'data' already exists 2024-08-19T11:24:22.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2024-08-19T11:24:23.609 INFO:tasks.workunit.client.0.smithi130.stderr:filesystem 'myfs' already exists 2024-08-19T11:24:23.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2024-08-19T11:24:28.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2024-08-19T11:24:28.999 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4155 and/or standby gids 2024-08-19T11:24:29.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2024-08-19T11:24:29.422 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4155 and/or standby gids 2024-08-19T11:24:29.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2024-08-19T11:24:30.034 INFO:tasks.workunit.client.0.smithi130.stderr:should be safe to stop [dne] 2024-08-19T11:24:30.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2024-08-19T11:24:30.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-08-19T11:24:30.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-08-19T11:24:30.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=b 2024-08-19T11:24:30.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-08-19T11:24:30.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/b 2024-08-19T11:24:30.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-08-19T11:24:30.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:24:30.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:24:30.094 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:24:30.094 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:24:30.094 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:24:30.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:24:30.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-08-19T11:24:30.095 INFO:tasks.workunit.client.0.smithi130.stdout:starting mds.b at 2024-08-19T11:24:30.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2024-08-19T11:24:35.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2024-08-19T11:24:35.702 INFO:tasks.workunit.client.0.smithi130.stderr:should be safe to stop [a] 2024-08-19T11:24:35.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2024-08-19T11:24:36.262 INFO:tasks.workunit.client.0.smithi130.stderr:should be safe to stop [b] 2024-08-19T11:24:36.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2024-08-19T11:24:36.653 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4155 and/or standby gids 4169 2024-08-19T11:24:36.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2024-08-19T11:24:37.213 INFO:tasks.workunit.client.0.smithi130.stderr:should be safe to stop [a,dne1,dne2] 2024-08-19T11:24:37.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2024-08-19T11:24:37.814 INFO:tasks.workunit.client.0.smithi130.stderr:should be safe to stop [b,dne] 2024-08-19T11:24:37.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2024-08-19T11:24:38.217 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4155 and/or standby gids 4169 2024-08-19T11:24:38.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2024-08-19T11:24:38.788 INFO:tasks.workunit.client.0.smithi130.stderr:should be safe to stop [dne1,dne2] 2024-08-19T11:24:38.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2024-08-19T11:24:38.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:24:38.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:24:38.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:24:38.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:24:38.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:24:38.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:24:38.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-08-19T11:24:38.913 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:24:38.913 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:24:38.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:24:38.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:24:38.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:24:39.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:24:39.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-08-19T11:24:39.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:24:39.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:24:39.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:24:39.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:24:39.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:24:39.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:24:39.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:24:39.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:24:39.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:24:39.235 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:24:39.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:24:39.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:24:39.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:24:39.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:24:39.237 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:24:39.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:24:39.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:24:39.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:24:39.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:24:39.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:24:39.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:24:39.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:24:39.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:24:39.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:24:39.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:24:39.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:24:39.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:24:39.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:24:39.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-08-19T11:24:39.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-08-19T11:24:39.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-08-19T11:24:39.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-08-19T11:24:39.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:24:39.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:24:39.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:24:39.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:24:39.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:24:39.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:24:39.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:24:39.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:24:39.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:24:39.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:24:39.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:24:39.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:24:39.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:24:39.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:24:39.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:24:39.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:24:39.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:24:39.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:24:39.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:24:39.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:24:39.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:24:39.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:24:39.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:24:39.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:24:39.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:24:39.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:24:39.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:24:39.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:24:39.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-08-19T11:24:39.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:24:39.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:24:39.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:24:39.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.48100 2024-08-19T11:24:39.259 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:24:39.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:24:39.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:24:39.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-08-19T11:24:39.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2024-08-19T11:24:39.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2024-08-19T11:24:39.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-08-19T11:24:39.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-08-19T11:24:39.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:24:39.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:24:39.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:24:39.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:24:39.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-08-19T11:24:39.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-08-19T11:24:39.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:24:39.778 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:24:39.778 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:24:39.778 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:24:39.778 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:24:39.779 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:24:39.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:24:39.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-08-19T11:24:39.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:24:39.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:24:39.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:24:39.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:24:39.997 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:24:39.997 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:24:39.998 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:24:39.998 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:24:39.998 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:24:39.998 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:24:39.998 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:24:39.998 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:24:39.998 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:24:39.998 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:24:39.999 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get fsid 2024-08-19T11:24:40.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:24:40.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:24:40.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:24:40.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:24:40.117 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:24:40.117 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:24:40.118 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:24:40.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:24:40.118 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:24:40.118 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:24:40.118 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:24:40.118 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:24:40.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:24:40.119 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get mon_host 2024-08-19T11:24:40.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2024-08-19T11:24:40.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-08-19T11:24:40.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:24:40.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:24:40.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:24:40.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-08-19T11:24:40.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:24:41.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:24:41.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:24:41.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:24:41.135 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:24:41.135 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:24:41.135 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:24:41.136 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:24:41.136 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:24:41.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:24:41.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2024-08-19T11:24:41.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-08-19T11:24:41.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:24:41.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:24:41.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:24:41.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-08-19T11:24:41.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-08-19T11:24:41.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:24:41.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:24:41.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:24:41.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-08-19T11:24:41.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-08-19T11:24:41.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:24:41.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:24:41.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-08-19T11:24:41.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:24:41.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:24:41.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:24:41.177 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:24:41.177 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:24:41.177 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:24:41.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:24:41.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:24:41.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=088d5b6f-4180-456f-be12-3650a6daf2fb 2024-08-19T11:24:41.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 088d5b6f-4180-456f-be12-3650a6daf2fb' 2024-08-19T11:24:41.180 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 088d5b6f-4180-456f-be12-3650a6daf2fb 2024-08-19T11:24:41.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:24:41.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5K8NmwQXNCxAA0E2PBplBjwQneZHwcL923g== 2024-08-19T11:24:41.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5K8NmwQXNCxAA0E2PBplBjwQneZHwcL923g=="}' 2024-08-19T11:24:41.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 088d5b6f-4180-456f-be12-3650a6daf2fb -i td/ok-to-stop/0/new.json 2024-08-19T11:24:41.517 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:24:41.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-08-19T11:24:41.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5K8NmwQXNCxAA0E2PBplBjwQneZHwcL923g== --osd-uuid 088d5b6f-4180-456f-be12-3650a6daf2fb 2024-08-19T11:24:41.566 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:24:41.566+0000 7feb1f69f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:24:41.594 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:24:41.598+0000 7feb1f69f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:24:41.597 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:24:41.598+0000 7feb1f69f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:24:41.597 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:24:41.598+0000 7feb1f69f7c0 -1 bdev(0x556d7fe39c00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:24:41.597 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:24:41.598+0000 7feb1f69f7c0 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-08-19T11:24:48.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-08-19T11:24:48.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:24:48.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:24:48.195 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:24:48.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:24:48.747 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:24:48.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:24:48.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:24:48.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:24:48.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:24:48.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:24:48.778 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:24:48.778+0000 7fef41ba47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:24:48.780 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:24:48.782+0000 7fef41ba47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:24:48.781 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:24:48.782+0000 7fef41ba47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:24:49.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:24:49.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:24:49.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:24:49.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:24:49.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:24:49.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:49.217 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:24:49.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:24:49.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:49.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:49.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:49.857 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:24:49.858+0000 7fef41ba47c0 -1 Falling back to public interface 2024-08-19T11:24:50.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:50.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:50.679 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:24:50.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:24:50.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:50.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:51.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:52.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:52.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:52.147 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:24:52.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:24:52.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:52.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:52.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:52.704 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:24:52.706+0000 7fef41ba47c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:24:53.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:53.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:53.571 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:24:53.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:24:53.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:53.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:54.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:55.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:55.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:55.139 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:24:55.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:24:55.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:55.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:55.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:56.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:56.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:56.763 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:24:56.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:24:56.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:56.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:57.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:58.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:58.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:58.204 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:24:58.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:24:58.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:58.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:24:58.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:24:59.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:24:59.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:24:59.769 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:24:59.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:24:59.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:24:59.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:25:00.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:01.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:01.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:01.206 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:25:01.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:25:01.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:01.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:25:01.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:02.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:02.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:02.720 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:25:02.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:25:02.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:02.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:25:03.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:04.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:04.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:04.193 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:25:04.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:25:04.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:04.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:25:04.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:05.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:05.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:05.712 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:25:05.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:25:05.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:05.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:25:06.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:07.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:07.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:07.232 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:25:07.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:25:07.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:07.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:25:08.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:09.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:09.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:09.153 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:25:09.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:25:09.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:09.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:25:10.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:11.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:11.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:11.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:25:11.043 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:25:11.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:11.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:25:11.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:12.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:12.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:12.914 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:25:12.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:25:12.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:12.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:25:13.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:14.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:14.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:14.811 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:25:14.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:25:14.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:14.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:25:15.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:16.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:16.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:16.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:25:16.927 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:25:16.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:16.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:25:17.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:18.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:18.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:18.646 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:25:18.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:25:18.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:18.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:25:19.097 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3757363891,v1:127.0.0.1:6803/3757363891] [v2:127.0.0.1:6804/3757363891,v1:127.0.0.1:6805/3757363891] exists,up 088d5b6f-4180-456f-be12-3650a6daf2fb 2024-08-19T11:25:19.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:25:19.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:25:19.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:25:19.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2024-08-19T11:25:19.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-08-19T11:25:19.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:25:19.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T11:25:19.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:25:19.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/1 2024-08-19T11:25:19.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-08-19T11:25:19.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:25:19.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:25:19.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:25:19.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2024-08-19T11:25:19.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2024-08-19T11:25:19.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:25:19.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:25:19.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-08-19T11:25:19.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:25:19.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:25:19.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:25:19.101 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:25:19.101 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:25:19.101 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:25:19.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:25:19.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:25:19.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:25:19.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:25:19.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:25:19.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-08-19T11:25:19.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-08-19T11:25:19.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:25:19.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:25:19.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:25:19.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:25:19.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:25:19.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:25:19.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/1 2024-08-19T11:25:19.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:25:19.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e216ddcb-66ca-4da4-8f4e-b1a069555335 2024-08-19T11:25:19.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e216ddcb-66ca-4da4-8f4e-b1a069555335' 2024-08-19T11:25:19.104 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 e216ddcb-66ca-4da4-8f4e-b1a069555335 2024-08-19T11:25:19.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:25:19.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCfK8NmB8A2BxAA59QLfu4yNm+KX09dqk2plg== 2024-08-19T11:25:19.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCfK8NmB8A2BxAA59QLfu4yNm+KX09dqk2plg=="}' 2024-08-19T11:25:19.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e216ddcb-66ca-4da4-8f4e-b1a069555335 -i td/ok-to-stop/1/new.json 2024-08-19T11:25:19.606 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:25:19.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/1/new.json 2024-08-19T11:25:19.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfK8NmB8A2BxAA59QLfu4yNm+KX09dqk2plg== --osd-uuid e216ddcb-66ca-4da4-8f4e-b1a069555335 2024-08-19T11:25:19.654 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:19.653+0000 7f8d015a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:25:19.686 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:19.685+0000 7f8d015a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:25:19.687 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:19.689+0000 7f8d015a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:25:19.687 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:19.689+0000 7f8d015a07c0 -1 bdev(0x55f4ca645c00 td/ok-to-stop/1/block) open stat got: (1) Operation not permitted 2024-08-19T11:25:19.688 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:19.689+0000 7f8d015a07c0 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2024-08-19T11:25:26.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/1/keyring 2024-08-19T11:25:26.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:25:26.495 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-08-19T11:25:26.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T11:25:26.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:25:27.052 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-08-19T11:25:27.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T11:25:27.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:25:27.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:25:27.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:25:27.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:25:27.082 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:27.081+0000 7efe450f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:25:27.105 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:27.105+0000 7efe450f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:25:27.106 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:27.109+0000 7efe450f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:25:27.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T11:25:27.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:25:27.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T11:25:27.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:25:27.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:25:27.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:27.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:25:27.500 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:25:27.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:27.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:27.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:28.317 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:28.317+0000 7efe450f47c0 -1 Falling back to public interface 2024-08-19T11:25:28.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:28.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:28.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:25:28.934 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:25:28.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:28.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:29.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:30.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:30.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:30.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:25:30.351 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:25:30.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:30.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:30.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:31.257 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:31.257+0000 7efe450f47c0 -1 osd.1 0 log_to_monitors true 2024-08-19T11:25:31.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:31.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:31.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:25:31.796 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:25:31.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:31.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:32.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:33.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:33.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:33.488 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:25:33.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:25:33.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:33.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:33.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:34.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:34.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:34.957 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:25:34.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:25:34.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:34.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:35.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:36.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:36.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:36.454 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:25:36.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:25:36.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:36.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:36.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:37.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:37.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:37.915 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:25:37.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:25:37.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:37.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:38.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:39.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:39.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:39.628 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:25:39.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:25:39.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:39.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:40.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:41.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:41.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:41.335 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:25:41.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:25:41.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:41.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:41.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:42.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:42.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:42.768 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:25:42.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:25:42.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:42.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:43.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:44.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:44.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:44.349 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:25:44.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:25:44.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:44.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:44.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:45.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:45.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:45.891 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:25:45.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:25:45.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:45.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:46.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:47.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:47.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:47.816 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:25:47.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:25:47.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:47.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:48.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:49.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:49.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:49.989 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:25:49.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:25:49.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:49.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:51.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:52.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:52.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:52.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:25:52.025 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:25:52.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:52.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:53.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:54.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:54.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:54.237 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:25:54.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:25:54.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:54.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:55.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:56.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:56.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:56.246 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:25:56.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:25:56.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:56.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:57.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:25:58.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:25:58.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:25:58.748 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:25:58.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:25:58.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:25:58.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:25:59.206 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2274233552,v1:127.0.0.1:6811/2274233552] [v2:127.0.0.1:6812/2274233552,v1:127.0.0.1:6813/2274233552] exists,up e216ddcb-66ca-4da4-8f4e-b1a069555335 2024-08-19T11:25:59.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:25:59.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:25:59.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:25:59.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/2 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:25:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-08-19T11:25:59.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:25:59.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:25:59.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:25:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:25:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:25:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:25:59.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:25:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:25:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:25:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:25:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:25:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-08-19T11:25:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-08-19T11:25:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:25:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:25:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:25:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:25:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:25:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:25:59.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/2 2024-08-19T11:25:59.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:25:59.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e41f5f5e-b0b0-423c-9ff7-8426bb59175a 2024-08-19T11:25:59.213 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 e41f5f5e-b0b0-423c-9ff7-8426bb59175a 2024-08-19T11:25:59.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e41f5f5e-b0b0-423c-9ff7-8426bb59175a' 2024-08-19T11:25:59.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:25:59.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDHK8NmJSmuDRAAI8gCdquiabX8TsxR6C0zZw== 2024-08-19T11:25:59.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDHK8NmJSmuDRAAI8gCdquiabX8TsxR6C0zZw=="}' 2024-08-19T11:25:59.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e41f5f5e-b0b0-423c-9ff7-8426bb59175a -i td/ok-to-stop/2/new.json 2024-08-19T11:25:59.804 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:25:59.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/2/new.json 2024-08-19T11:25:59.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDHK8NmJSmuDRAAI8gCdquiabX8TsxR6C0zZw== --osd-uuid e41f5f5e-b0b0-423c-9ff7-8426bb59175a 2024-08-19T11:25:59.855 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:59.856+0000 7f649e1c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:25:59.895 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:59.896+0000 7f649e1c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:25:59.897 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:59.896+0000 7f649e1c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:25:59.898 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:59.896+0000 7f649e1c27c0 -1 bdev(0x560b69a7e800 td/ok-to-stop/2/block) open stat got: (1) Operation not permitted 2024-08-19T11:25:59.898 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:25:59.896+0000 7f649e1c27c0 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2024-08-19T11:26:06.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/2/keyring 2024-08-19T11:26:06.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:26:06.699 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-08-19T11:26:06.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T11:26:06.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:26:07.258 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-08-19T11:26:07.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T11:26:07.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:26:07.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:26:07.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:26:07.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:26:07.283 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:07.284+0000 7f9588df97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:26:07.286 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:07.284+0000 7f9588df97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:26:07.287 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:07.288+0000 7f9588df97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:26:07.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T11:26:07.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:26:07.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T11:26:07.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:26:07.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:26:07.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:07.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:26:07.721 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:26:07.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:07.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:08.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:08.277 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:08.276+0000 7f9588df97c0 -1 Falling back to public interface 2024-08-19T11:26:09.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:09.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:09.154 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:26:09.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:26:09.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:09.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:09.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:10.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:10.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:10.577 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:26:10.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:26:10.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:10.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:11.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:11.145 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:11.144+0000 7f9588df97c0 -1 osd.2 0 log_to_monitors true 2024-08-19T11:26:12.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:12.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:12.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:26:12.012 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:26:12.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:12.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:12.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:13.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:13.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:13.572 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:26:13.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:26:13.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:13.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:14.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:15.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:15.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:15.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:26:15.244 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:26:15.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:15.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:15.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:16.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:16.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:16.707 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:26:16.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:26:16.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:16.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:17.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:18.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:18.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:18.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:26:18.276 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:26:18.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:18.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:18.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:19.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:19.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:19.769 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:26:19.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:26:19.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:19.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:20.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:21.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:21.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:21.359 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:26:21.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:26:21.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:21.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:21.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:22.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:22.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:22.798 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:26:22.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:26:22.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:22.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:23.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:24.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:24.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:24.347 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:26:24.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:26:24.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:24.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:24.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:25.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:25.906 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:26:25.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:25.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:26:25.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:25.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:26.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:27.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:27.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:27.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:26:27.861 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:26:27.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:27.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:28.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:29.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:29.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:29.754 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:26:29.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:26:29.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:29.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:30.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:31.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:31.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:31.883 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:26:31.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:26:31.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:31.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:33.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:34.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:34.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:34.079 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:26:34.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:26:34.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:34.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:34.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:35.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:35.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:35.901 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:26:35.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:26:35.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:35.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:36.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:37.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:37.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:37.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:26:37.684 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:26:37.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:37.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:38.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:39.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:39.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:39.300 INFO:tasks.workunit.client.0.smithi130.stdout:19 2024-08-19T11:26:39.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T11:26:39.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:39.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:26:39.722 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2388650577,v1:127.0.0.1:6819/2388650577] [v2:127.0.0.1:6820/2388650577,v1:127.0.0.1:6821/2388650577] exists,up e41f5f5e-b0b0-423c-9ff7-8426bb59175a 2024-08-19T11:26:39.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:26:39.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:26:39.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:26:39.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2024-08-19T11:26:39.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-08-19T11:26:39.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:26:39.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-08-19T11:26:39.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:26:39.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/3 2024-08-19T11:26:39.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-08-19T11:26:39.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:26:39.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:26:39.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:26:39.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2024-08-19T11:26:39.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2024-08-19T11:26:39.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:26:39.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:26:39.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-08-19T11:26:39.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:26:39.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:26:39.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:26:39.726 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:26:39.726 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:26:39.726 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:26:39.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:26:39.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:26:39.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:26:39.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:26:39.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:26:39.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-08-19T11:26:39.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-08-19T11:26:39.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:26:39.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:26:39.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:26:39.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:26:39.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:26:39.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:26:39.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/3 2024-08-19T11:26:39.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:26:39.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=44c6faf8-9654-46d2-9628-e4fd8a10fc07 2024-08-19T11:26:39.728 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 44c6faf8-9654-46d2-9628-e4fd8a10fc07 2024-08-19T11:26:39.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 44c6faf8-9654-46d2-9628-e4fd8a10fc07' 2024-08-19T11:26:39.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:26:39.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvK8Nm4TK8LBAA8YfJ+gDZNH2fT5LnASvqtg== 2024-08-19T11:26:39.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvK8Nm4TK8LBAA8YfJ+gDZNH2fT5LnASvqtg=="}' 2024-08-19T11:26:39.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 44c6faf8-9654-46d2-9628-e4fd8a10fc07 -i td/ok-to-stop/3/new.json 2024-08-19T11:26:40.213 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:26:40.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/3/new.json 2024-08-19T11:26:40.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvK8Nm4TK8LBAA8YfJ+gDZNH2fT5LnASvqtg== --osd-uuid 44c6faf8-9654-46d2-9628-e4fd8a10fc07 2024-08-19T11:26:40.262 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:40.259+0000 7ff5f9d057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:26:40.298 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:40.295+0000 7ff5f9d057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:26:40.300 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:40.299+0000 7ff5f9d057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:26:40.300 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:40.299+0000 7ff5f9d057c0 -1 bdev(0x5567fa9cdc00 td/ok-to-stop/3/block) open stat got: (1) Operation not permitted 2024-08-19T11:26:40.300 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:40.299+0000 7ff5f9d057c0 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2024-08-19T11:26:47.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/3/keyring 2024-08-19T11:26:47.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:26:47.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-08-19T11:26:47.243 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-08-19T11:26:47.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:26:47.816 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-08-19T11:26:47.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-08-19T11:26:47.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:26:47.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:26:47.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:26:47.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:26:47.846 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:47.843+0000 7ff004fc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:26:47.868 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:47.867+0000 7ff004fc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:26:47.869 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:47.867+0000 7ff004fc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:26:48.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-08-19T11:26:48.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:26:48.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-19T11:26:48.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:26:48.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:26:48.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:48.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:26:48.268 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:26:48.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:48.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:26:48.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:49.098 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:49.095+0000 7ff004fc47c0 -1 Falling back to public interface 2024-08-19T11:26:49.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:49.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:49.694 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:26:49.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:26:49.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:49.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:26:50.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:51.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:51.125 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:26:51.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:51.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:26:51.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:51.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:26:51.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:52.160 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:26:52.159+0000 7ff004fc47c0 -1 osd.3 0 log_to_monitors true 2024-08-19T11:26:52.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:52.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:52.556 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:26:52.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:26:52.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:52.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:26:53.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:54.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:54.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:54.089 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:26:54.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:26:54.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:54.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:26:54.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:55.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:55.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:55.684 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:26:55.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:26:55.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:55.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:26:56.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:57.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:57.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:57.358 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:26:57.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:26:57.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:57.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:26:57.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:26:58.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:26:58.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:26:58.790 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:26:58.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:26:58.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:26:58.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:26:59.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:27:00.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:27:00.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:27:00.366 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:27:00.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:27:00.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:27:00.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:27:00.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:27:01.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:27:01.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:27:01.871 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:27:01.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:27:01.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:27:01.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:27:02.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:27:03.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:27:03.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:27:03.631 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:27:03.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:27:03.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:27:03.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:27:04.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:27:05.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:27:05.194 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:27:05.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:27:05.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:27:05.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:27:05.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:27:05.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:27:06.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:27:06.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:27:06.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:27:06.630 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:27:06.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:27:06.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:27:07.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:27:08.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:27:08.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:27:08.159 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:27:08.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:27:08.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:27:08.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:27:09.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:27:10.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:27:10.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:27:10.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:27:10.274 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:27:10.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:27:10.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:27:11.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:27:12.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:27:12.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:27:12.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:27:12.518 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:27:12.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:27:12.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:27:13.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:27:14.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:27:14.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:27:14.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:27:14.609 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:27:14.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:27:14.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:27:15.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:27:16.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:27:16.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:27:16.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:27:16.504 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:27:16.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:27:16.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:27:17.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:27:18.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:27:18.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:27:18.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:27:18.348 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:27:18.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:27:18.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-19T11:27:18.806 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/877106572,v1:127.0.0.1:6827/877106572] [v2:127.0.0.1:6828/877106572,v1:127.0.0.1:6829/877106572] exists,up 44c6faf8-9654-46d2-9628-e4fd8a10fc07 2024-08-19T11:27:18.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:27:18.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:27:18.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:27:18.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-08-19T11:27:19.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2024-08-19T11:27:20.385 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'ec' already exists 2024-08-19T11:27:20.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2024-08-19T11:27:20.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:27:20.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:27:20.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:27:20.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:27:20.403 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:27:20.403 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:27:20.404 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:27:20.404 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:27:20.404 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:27:20.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:27:20.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:27:20.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:27:20.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:27:20.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:27:20.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:27:21.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-19T11:27:21.407 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-08-19T11:27:21.407 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-08-19T11:27:21.407 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-08-19T11:27:21.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:27:21.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:21.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:27:21.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836508 2024-08-19T11:27:21.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836508 2024-08-19T11:27:21.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836508' 2024-08-19T11:27:21.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:21.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T11:27:21.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672983 2024-08-19T11:27:21.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672983 2024-08-19T11:27:21.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836508 1-42949672983' 2024-08-19T11:27:21.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:21.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T11:27:22.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509458 2024-08-19T11:27:22.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509458 2024-08-19T11:27:22.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836508 1-42949672983 2-64424509458' 2024-08-19T11:27:22.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:22.119 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T11:27:22.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=85899345929 2024-08-19T11:27:22.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 85899345929 2024-08-19T11:27:22.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836508 1-42949672983 2-64424509458 3-85899345929' 2024-08-19T11:27:22.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:22.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836508 2024-08-19T11:27:22.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:22.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:27:22.400 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836508 2024-08-19T11:27:22.400 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:22.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836508 2024-08-19T11:27:22.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-08-19T11:27:22.403 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836508 2024-08-19T11:27:22.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:27:22.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836508 -lt 21474836508 2024-08-19T11:27:22.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:22.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672983 2024-08-19T11:27:22.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:22.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-19T11:27:22.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672983 2024-08-19T11:27:22.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:22.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672983 2024-08-19T11:27:22.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-08-19T11:27:22.918 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672983 2024-08-19T11:27:22.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T11:27:23.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672981 -lt 42949672983 2024-08-19T11:27:23.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:27:24.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:27:24.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T11:27:24.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672984 -lt 42949672983 2024-08-19T11:27:24.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:24.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509458 2024-08-19T11:27:24.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:24.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-19T11:27:24.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509458 2024-08-19T11:27:24.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:24.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509458 2024-08-19T11:27:24.925 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509458 2024-08-19T11:27:24.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-08-19T11:27:24.926 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T11:27:25.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509459 -lt 64424509458 2024-08-19T11:27:25.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:25.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345929 2024-08-19T11:27:25.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:25.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=3 2024-08-19T11:27:25.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 3-85899345929 2024-08-19T11:27:25.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:25.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=85899345929 2024-08-19T11:27:25.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2024-08-19T11:27:25.422 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345929 2024-08-19T11:27:25.422 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T11:27:25.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 85899345929 -lt 85899345929 2024-08-19T11:27:25.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:27:25.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:27:25.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:27:26.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-08-19T11:27:26.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:27:26.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:27:26.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:27:26.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:27:26.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:27:26.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:27:26.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:27:26.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-08-19T11:27:26.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:27:26.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:27:26.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:27:27.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-08-19T11:27:27.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:27:27.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:27:27.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2024-08-19T11:27:28.043 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 min_size to 3 2024-08-19T11:27:28.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2024-08-19T11:27:28.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:27:28.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:27:28.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:27:28.066 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:27:28.067 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:27:28.067 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:27:28.067 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:27:28.067 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:27:28.067 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:27:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:27:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:27:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:27:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:27:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:27:28.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:27:28.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-19T11:27:28.624 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-08-19T11:27:28.624 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-08-19T11:27:28.624 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-08-19T11:27:28.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:27:28.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:28.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:27:28.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836511 2024-08-19T11:27:28.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836511 2024-08-19T11:27:28.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836511' 2024-08-19T11:27:28.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:28.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T11:27:28.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672987 2024-08-19T11:27:28.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672987 2024-08-19T11:27:28.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836511 1-42949672987' 2024-08-19T11:27:28.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:28.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T11:27:29.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509462 2024-08-19T11:27:29.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509462 2024-08-19T11:27:29.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836511 1-42949672987 2-64424509462' 2024-08-19T11:27:29.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:29.120 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T11:27:29.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=85899345932 2024-08-19T11:27:29.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 85899345932 2024-08-19T11:27:29.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836511 1-42949672987 2-64424509462 3-85899345932' 2024-08-19T11:27:29.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:29.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836511 2024-08-19T11:27:29.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:29.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:27:29.271 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836511 2024-08-19T11:27:29.271 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:29.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836511 2024-08-19T11:27:29.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-08-19T11:27:29.272 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836511 2024-08-19T11:27:29.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:27:29.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836509 -lt 21474836511 2024-08-19T11:27:29.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:27:30.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:27:30.666 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:27:31.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836512 -lt 21474836511 2024-08-19T11:27:31.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:31.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672987 2024-08-19T11:27:31.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:31.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-19T11:27:31.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672987 2024-08-19T11:27:31.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:31.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672987 2024-08-19T11:27:31.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-08-19T11:27:31.095 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672987 2024-08-19T11:27:31.096 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T11:27:31.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672987 -lt 42949672987 2024-08-19T11:27:31.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:31.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:31.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509462 2024-08-19T11:27:31.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-19T11:27:31.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509462 2024-08-19T11:27:31.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:31.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509462 2024-08-19T11:27:31.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-08-19T11:27:31.515 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509462 2024-08-19T11:27:31.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T11:27:31.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509462 -lt 64424509462 2024-08-19T11:27:31.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:31.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345932 2024-08-19T11:27:31.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:31.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=3 2024-08-19T11:27:31.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 3-85899345932 2024-08-19T11:27:31.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:31.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=85899345932 2024-08-19T11:27:31.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-08-19T11:27:31.938 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345932 2024-08-19T11:27:31.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T11:27:32.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 85899345933 -lt 85899345932 2024-08-19T11:27:32.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:27:32.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:27:32.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:27:32.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-08-19T11:27:32.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:27:32.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:27:32.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:27:32.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:27:32.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:27:32.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:27:32.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:27:33.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-08-19T11:27:33.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:27:33.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:27:33.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:27:33.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-08-19T11:27:33.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:27:33.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:27:33.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2024-08-19T11:27:34.382 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:34.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2024-08-19T11:27:34.778 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:34.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2024-08-19T11:27:35.187 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:35.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2024-08-19T11:27:35.579 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:35.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-08-19T11:27:35.895 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:35.896 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-08-19T11:27:35.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-08-19T11:27:36.190 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:36.191 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-08-19T11:27:36.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-08-19T11:27:36.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2024-08-19T11:27:36.661 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:36.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2024-08-19T11:27:36.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2024-08-19T11:27:37.072 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:37.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2024-08-19T11:27:38.111 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 min_size to 2 2024-08-19T11:27:38.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2024-08-19T11:27:38.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:27:38.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:27:38.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:27:38.132 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:27:38.132 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:27:38.132 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:27:38.133 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:27:38.133 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:27:38.133 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:27:38.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:27:38.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:27:38.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:27:38.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:27:38.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:27:38.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:27:39.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-19T11:27:39.050 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-08-19T11:27:39.050 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-08-19T11:27:39.050 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-08-19T11:27:39.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:27:39.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:39.051 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:27:39.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836515 2024-08-19T11:27:39.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836515 2024-08-19T11:27:39.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836515' 2024-08-19T11:27:39.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:39.203 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T11:27:39.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672991 2024-08-19T11:27:39.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672991 2024-08-19T11:27:39.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836515 1-42949672991' 2024-08-19T11:27:39.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:39.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T11:27:39.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509466 2024-08-19T11:27:39.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509466 2024-08-19T11:27:39.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836515 1-42949672991 2-64424509466' 2024-08-19T11:27:39.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:39.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T11:27:39.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=85899345936 2024-08-19T11:27:39.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 85899345936 2024-08-19T11:27:39.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836515 1-42949672991 2-64424509466 3-85899345936' 2024-08-19T11:27:39.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:39.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836515 2024-08-19T11:27:39.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:39.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:27:39.720 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836515 2024-08-19T11:27:39.720 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:39.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836515 2024-08-19T11:27:39.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-08-19T11:27:39.721 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836515 2024-08-19T11:27:39.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:27:40.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836516 -lt 21474836515 2024-08-19T11:27:40.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:40.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672991 2024-08-19T11:27:40.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:40.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-19T11:27:40.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672991 2024-08-19T11:27:40.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:40.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672991 2024-08-19T11:27:40.147 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672991 2024-08-19T11:27:40.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-08-19T11:27:40.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T11:27:40.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672991 -lt 42949672991 2024-08-19T11:27:40.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:40.620 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509466 2024-08-19T11:27:40.620 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:40.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-19T11:27:40.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509466 2024-08-19T11:27:40.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:40.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509466 2024-08-19T11:27:40.624 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509466 2024-08-19T11:27:40.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-08-19T11:27:40.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T11:27:41.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509466 -lt 64424509466 2024-08-19T11:27:41.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:41.087 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345936 2024-08-19T11:27:41.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:41.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=3 2024-08-19T11:27:41.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 3-85899345936 2024-08-19T11:27:41.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:41.091 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345936 2024-08-19T11:27:41.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=85899345936 2024-08-19T11:27:41.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.3 seq 85899345936' 2024-08-19T11:27:41.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T11:27:41.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 85899345937 -lt 85899345936 2024-08-19T11:27:41.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:27:41.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:27:41.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:27:42.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-08-19T11:27:42.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:27:42.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:27:42.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:27:42.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:27:42.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:27:42.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:27:42.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:27:42.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=11 2024-08-19T11:27:42.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:27:42.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:27:42.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:27:43.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 11 = 32 2024-08-19T11:27:43.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 11 '!=' -1 2024-08-19T11:27:43.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T11:27:43.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=11 2024-08-19T11:27:43.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T11:27:43.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T11:27:43.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T11:27:43.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:27:43.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:27:43.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:27:43.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:27:43.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:27:43.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:27:43.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:27:43.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=11 2024-08-19T11:27:43.608 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:27:43.608 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:27:43.608 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:27:44.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 11 = 32 2024-08-19T11:27:44.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 11 '!=' 11 2024-08-19T11:27:44.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-19T11:27:44.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T11:27:44.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T11:27:44.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T11:27:44.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T11:27:44.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T11:27:44.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T11:27:44.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T11:27:44.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T11:27:44.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-19T11:27:44.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T11:27:44.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-19T11:27:44.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T11:27:44.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:27:44.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:27:44.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:27:44.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:27:44.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:27:44.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:27:44.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:27:45.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=24 2024-08-19T11:27:45.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:27:45.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:27:45.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:27:45.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 24 = 32 2024-08-19T11:27:45.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 24 '!=' 11 2024-08-19T11:27:45.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-19T11:27:45.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=24 2024-08-19T11:27:45.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-19T11:27:45.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-19T11:27:45.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-19T11:27:45.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:27:45.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:27:45.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:27:45.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:27:45.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:27:45.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:27:45.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:27:46.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-08-19T11:27:46.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:27:46.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:27:46.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:27:46.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-08-19T11:27:46.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:27:46.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:27:46.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-08-19T11:27:47.351 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:47.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-08-19T11:27:47.764 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:47.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-08-19T11:27:48.077 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:48.078 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-08-19T11:27:48.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-08-19T11:27:48.360 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:48.361 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-08-19T11:27:48.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-08-19T11:27:48.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2024-08-19T11:27:48.757 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:48.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2024-08-19T11:27:48.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2024-08-19T11:27:49.160 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:49.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2024-08-19T11:27:49.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2024-08-19T11:27:49.594 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:49.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2024-08-19T11:27:49.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2024-08-19T11:27:50.009 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:50.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2024-08-19T11:27:50.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:27:50.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:27:50.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:27:50.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:27:50.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:27:51.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:27:51.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2024-08-19T11:27:51.884 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already down. 2024-08-19T11:27:51.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2024-08-19T11:27:51.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-08-19T11:27:51.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-08-19T11:27:51.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-08-19T11:27:51.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-08-19T11:27:51.904 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:27:51.904 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:27:51.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:27:51.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:27:51.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:27:52.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:27:52.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-08-19T11:27:52.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-08-19T11:27:52.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-08-19T11:27:52.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:27:52.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:27:52.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-19T11:27:52.512 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-08-19T11:27:52.512 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-08-19T11:27:52.512 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-08-19T11:27:52.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:27:52.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:52.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:27:52.645 INFO:tasks.workunit.client.0.smithi130.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-08-19T11:27:52.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq= 2024-08-19T11:27:52.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z '' 2024-08-19T11:27:52.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: continue 2024-08-19T11:27:52.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:52.647 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T11:27:52.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672995 2024-08-19T11:27:52.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672995 2024-08-19T11:27:52.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 1-42949672995' 2024-08-19T11:27:52.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:52.794 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-19T11:27:52.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509470 2024-08-19T11:27:52.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509470 2024-08-19T11:27:52.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 1-42949672995 2-64424509470' 2024-08-19T11:27:52.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:27:52.945 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-19T11:27:53.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=85899345941 2024-08-19T11:27:53.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 85899345941 2024-08-19T11:27:53.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 1-42949672995 2-64424509470 3-85899345941' 2024-08-19T11:27:53.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:53.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672995 2024-08-19T11:27:53.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:53.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-19T11:27:53.095 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672995 2024-08-19T11:27:53.095 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:53.096 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672995 2024-08-19T11:27:53.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672995 2024-08-19T11:27:53.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2024-08-19T11:27:53.097 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T11:27:53.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672993 -lt 42949672995 2024-08-19T11:27:53.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:27:54.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:27:54.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T11:27:54.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672996 -lt 42949672995 2024-08-19T11:27:54.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:54.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509470 2024-08-19T11:27:54.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:54.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-19T11:27:54.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509470 2024-08-19T11:27:54.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:54.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509470 2024-08-19T11:27:54.929 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509470 2024-08-19T11:27:54.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-08-19T11:27:54.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-19T11:27:55.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509471 -lt 64424509470 2024-08-19T11:27:55.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:27:55.362 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345941 2024-08-19T11:27:55.362 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:27:55.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=3 2024-08-19T11:27:55.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 3-85899345941 2024-08-19T11:27:55.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:27:55.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=85899345941 2024-08-19T11:27:55.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2024-08-19T11:27:55.365 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345941 2024-08-19T11:27:55.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-19T11:27:55.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 85899345941 -lt 85899345941 2024-08-19T11:27:55.869 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-08-19T11:27:55.869 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:27:55.869 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:27:56.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 32 == 0 2024-08-19T11:27:56.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-08-19T11:27:56.410 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-08-19T11:27:56.410 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-08-19T11:27:56.410 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-08-19T11:27:56.410 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-08-19T11:27:56.410 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-08-19T11:27:56.410 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-08-19T11:27:56.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=32 2024-08-19T11:27:56.855 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-08-19T11:27:56.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:27:56.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:27:57.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 32 = 32 2024-08-19T11:27:57.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-08-19T11:27:57.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-08-19T11:27:57.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2024-08-19T11:27:57.813 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2024-08-19T11:27:57.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-08-19T11:27:58.227 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:58.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-08-19T11:27:58.553 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:58.553 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-08-19T11:27:58.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-08-19T11:27:58.835 INFO:tasks.workunit.client.0.smithi130.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-08-19T11:27:58.835 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-08-19T11:27:58.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-08-19T11:27:58.840 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:27:58.840 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:27:58.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:27:58.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:27:58.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:27:59.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:27:59.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-08-19T11:27:59.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:27:59.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:27:59.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:27:59.154 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:27:59.154 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:27:59.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:27:59.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:27:59.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:27:59.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:27:59.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:27:59.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:27:59.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:27:59.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:27:59.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:27:59.164 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:27:59.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:27:59.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:27:59.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:27:59.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:27:59.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:27:59.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:27:59.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:27:59.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:27:59.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:27:59.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:27:59.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:27:59.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:27:59.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:27:59.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-08-19T11:27:59.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-08-19T11:27:59.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-08-19T11:27:59.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-08-19T11:27:59.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:27:59.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:27:59.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:27:59.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:27:59.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:27:59.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:27:59.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:27:59.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:27:59.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:27:59.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:27:59.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:27:59.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:27:59.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:27:59.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:27:59.196 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:27:59.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:27:59.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:27:59.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:27:59.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:27:59.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:27:59.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:27:59.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:27:59.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:27:59.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:27:59.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:27:59.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:27:59.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:27:59.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:27:59.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-08-19T11:27:59.201 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:27:59.201 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:27:59.202 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:27:59.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.48100 2024-08-19T11:27:59.203 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:27:59.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:27:59.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:27:59.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-08-19T11:27:59.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2024-08-19T11:27:59.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2024-08-19T11:27:59.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-08-19T11:27:59.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-08-19T11:27:59.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:27:59.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:27:59.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:27:59.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:27:59.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-08-19T11:27:59.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-08-19T11:27:59.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:27:59.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:27:59.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:27:59.677 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:27:59.677 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:27:59.677 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:27:59.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:27:59.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-08-19T11:27:59.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:27:59.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:27:59.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:27:59.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:27:59.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:27:59.890 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:27:59.890 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:27:59.890 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:27:59.890 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:27:59.890 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:27:59.891 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:27:59.891 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:27:59.891 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:27:59.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:27:59.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get fsid 2024-08-19T11:28:00.006 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:28:00.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:00.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:28:00.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:28:00.009 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:28:00.009 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:28:00.009 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:28:00.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:28:00.009 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:00.009 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:00.009 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:00.010 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:28:00.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:00.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get mon_host 2024-08-19T11:28:00.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2024-08-19T11:28:00.946 INFO:tasks.workunit.client.0.smithi130.stderr:invalid mon(s) specified: dne 2024-08-19T11:28:00.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2024-08-19T11:28:01.124 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available () after stopping mons a 2024-08-19T11:28:01.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2024-08-19T11:28:01.322 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-08-19T11:28:01.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2024-08-19T11:28:01.496 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: removing mon.a would break quorum 2024-08-19T11:28:01.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2024-08-19T11:28:01.744 INFO:tasks.workunit.client.0.smithi130.stderr:mon.dne does not exist 2024-08-19T11:28:01.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-08-19T11:28:01.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:28:01.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:28:01.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:28:01.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:28:01.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:28:01.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:28:01.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-08-19T11:28:01.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:28:01.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:28:01.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:28:01.867 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:28:01.867 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:28:01.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:28:01.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:28:01.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:28:01.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:28:01.872 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:28:01.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:28:01.873 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:28:01.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:28:01.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:28:01.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:28:01.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:28:01.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:28:01.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:28:01.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:28:01.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:28:01.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:28:01.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:28:01.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:28:01.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:01.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:01.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:28:01.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:28:01.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:28:01.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-08-19T11:28:01.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-08-19T11:28:01.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-08-19T11:28:01.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-08-19T11:28:01.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:28:01.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:28:01.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:28:01.883 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:28:01.883 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:28:01.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:28:01.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:28:01.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:28:01.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:28:01.885 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:28:01.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:28:01.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:28:01.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:28:01.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:28:01.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:28:01.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:28:01.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:28:01.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:28:01.890 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:28:01.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:28:01.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:28:01.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:28:01.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:28:01.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:01.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:01.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:28:01.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:28:01.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:28:01.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-08-19T11:28:01.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:28:01.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:01.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:01.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.48100 2024-08-19T11:28:01.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:28:01.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:28:01.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:28:01.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-08-19T11:28:01.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2024-08-19T11:28:01.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2024-08-19T11:28:01.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2024-08-19T11:28:01.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-08-19T11:28:01.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:28:01.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:28:01.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:28:01.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:28:01.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-08-19T11:28:01.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-08-19T11:28:02.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:28:02.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:28:02.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:28:02.377 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:28:02.377 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:02.377 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:02.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:28:02.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-08-19T11:28:02.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:28:02.596 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:28:02.596 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:02.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:28:02.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:28:02.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:28:02.598 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:28:02.598 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:28:02.598 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:28:02.598 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:02.598 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:02.598 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:02.599 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:28:02.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:02.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get fsid 2024-08-19T11:28:02.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:28:02.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:02.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:28:02.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:28:02.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:28:02.718 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:28:02.718 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:28:02.718 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:28:02.718 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:02.718 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:02.718 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:02.718 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:28:02.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:02.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get mon_host 2024-08-19T11:28:02.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-08-19T11:28:02.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:28:02.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:28:02.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-08-19T11:28:02.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:28:02.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-08-19T11:28:02.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-08-19T11:28:03.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:28:03.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:28:03.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:28:03.338 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:28:03.339 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:03.339 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:03.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:28:03.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-08-19T11:28:03.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:28:03.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-08-19T11:28:03.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:03.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-08-19T11:28:03.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:28:03.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:28:03.583 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-08-19T11:28:03.583 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-08-19T11:28:03.583 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-08-19T11:28:03.583 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:03.583 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:03.583 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:03.583 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.b.asok 2024-08-19T11:28:03.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:03.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.b.asok config get fsid 2024-08-19T11:28:03.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-08-19T11:28:03.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:03.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-08-19T11:28:03.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:28:03.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:28:03.703 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-08-19T11:28:03.703 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-08-19T11:28:03.703 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-08-19T11:28:03.703 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:03.703 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:03.703 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:03.703 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.b.asok 2024-08-19T11:28:03.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:03.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.b.asok config get mon_host 2024-08-19T11:28:03.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2024-08-19T11:28:07.048 INFO:tasks.workunit.client.0.smithi130.stderr:invalid mon(s) specified: dne 2024-08-19T11:28:07.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2024-08-19T11:28:07.330 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons a 2024-08-19T11:28:07.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2024-08-19T11:28:07.567 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons b 2024-08-19T11:28:07.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2024-08-19T11:28:07.867 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available () after stopping mons a b 2024-08-19T11:28:07.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2024-08-19T11:28:08.151 INFO:tasks.workunit.client.0.smithi130.stderr:adding another mon that is not yet online will not break quorum 2024-08-19T11:28:08.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2024-08-19T11:28:08.396 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.a 2024-08-19T11:28:08.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2024-08-19T11:28:08.629 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.b 2024-08-19T11:28:08.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2024-08-19T11:28:08.975 INFO:tasks.workunit.client.0.smithi130.stderr:mon.dne does not exist 2024-08-19T11:28:08.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-08-19T11:28:08.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:28:08.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:28:08.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:28:08.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:28:08.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:28:09.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:28:09.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-08-19T11:28:09.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:28:09.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:28:09.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:28:09.105 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:28:09.105 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:28:09.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:28:09.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:28:09.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:28:09.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:28:09.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:28:09.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:28:09.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:28:09.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:28:09.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:28:09.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:28:09.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:28:09.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:28:09.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:28:09.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:28:09.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:28:09.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:28:09.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:28:09.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:28:09.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:09.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:09.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:28:09.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:28:09.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:28:09.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-08-19T11:28:09.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-08-19T11:28:09.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-08-19T11:28:09.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-08-19T11:28:09.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:28:09.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:28:09.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:28:09.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:28:09.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:28:09.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:28:09.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:28:09.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:28:09.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:28:09.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:28:09.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:28:09.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:28:09.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:28:09.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:28:09.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:28:09.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:28:09.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:28:09.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:28:09.132 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:28:09.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:28:09.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:28:09.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:28:09.136 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:28:09.136 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:09.136 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:09.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:28:09.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:28:09.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:28:09.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-08-19T11:28:09.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:28:09.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:09.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:09.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.48100 2024-08-19T11:28:09.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:28:09.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:28:09.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:28:09.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-08-19T11:28:09.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2024-08-19T11:28:09.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2024-08-19T11:28:09.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2024-08-19T11:28:09.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-08-19T11:28:09.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:28:09.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:28:09.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:28:09.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:28:09.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-08-19T11:28:09.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-08-19T11:28:09.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:28:09.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:28:09.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:28:09.624 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:28:09.624 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:09.624 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:09.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:28:09.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-08-19T11:28:09.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:28:09.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:28:09.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:09.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:28:09.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:28:09.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:28:09.847 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:28:09.847 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:28:09.847 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:28:09.847 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:09.847 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:09.847 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:09.847 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:28:09.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:09.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get fsid 2024-08-19T11:28:09.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:28:09.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:09.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:28:09.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:28:09.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:28:09.968 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:28:09.968 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:28:09.968 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:28:09.968 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:09.968 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:09.968 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:09.968 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:28:09.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:09.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get mon_host 2024-08-19T11:28:10.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-08-19T11:28:10.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:28:10.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:28:10.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-08-19T11:28:10.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:28:10.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-08-19T11:28:10.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-08-19T11:28:10.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:28:10.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:28:10.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:28:10.570 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:28:10.570 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:10.570 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:10.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:28:10.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-08-19T11:28:10.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:28:10.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-08-19T11:28:10.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:10.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-08-19T11:28:10.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:28:10.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:28:10.801 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-08-19T11:28:10.801 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-08-19T11:28:10.801 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-08-19T11:28:10.801 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:10.801 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:10.801 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:10.801 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.b.asok 2024-08-19T11:28:10.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:10.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.b.asok config get fsid 2024-08-19T11:28:10.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-08-19T11:28:10.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:10.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-08-19T11:28:10.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:28:10.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:28:10.920 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-08-19T11:28:10.920 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-08-19T11:28:10.920 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-08-19T11:28:10.920 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:10.920 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:10.920 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:10.920 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.b.asok 2024-08-19T11:28:10.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:10.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.b.asok config get mon_host 2024-08-19T11:28:11.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-08-19T11:28:11.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:28:11.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:28:11.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-08-19T11:28:11.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:28:11.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-08-19T11:28:11.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-08-19T11:28:11.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:28:11.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:28:11.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:28:11.566 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:28:11.566 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:11.566 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:11.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:28:11.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-08-19T11:28:11.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:28:11.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-08-19T11:28:11.778 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:11.778 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-08-19T11:28:11.778 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:28:11.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:28:11.779 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-08-19T11:28:11.779 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-08-19T11:28:11.779 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-08-19T11:28:11.779 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:11.779 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:11.779 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:11.780 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.c.asok 2024-08-19T11:28:11.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:11.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.c.asok config get fsid 2024-08-19T11:28:11.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-08-19T11:28:11.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:11.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-08-19T11:28:11.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:28:11.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:28:11.898 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-08-19T11:28:11.899 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-08-19T11:28:11.899 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-08-19T11:28:11.899 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:11.899 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:11.899 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:11.899 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.c.asok 2024-08-19T11:28:11.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:11.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.c.asok config get mon_host 2024-08-19T11:28:12.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2024-08-19T11:28:12.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-08-19T11:28:12.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-08-19T11:28:12.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-08-19T11:28:12.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-08-19T11:28:12.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-08-19T11:28:12.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-08-19T11:28:12.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1724066952 2024-08-19T11:28:12.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:12.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066892 -lt 1724066952 ]] 2024-08-19T11:28:12.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:28:12.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:28:18.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:28:18.256 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:18.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:18.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:28:18.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-08-19T11:28:18.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-08-19T11:28:18.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-08-19T11:28:18.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-08-19T11:28:18.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-08-19T11:28:18.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2024-08-19T11:28:21.522 INFO:tasks.workunit.client.0.smithi130.stderr:invalid mon(s) specified: dne 2024-08-19T11:28:21.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-08-19T11:28:21.834 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (b,c) after stopping a 2024-08-19T11:28:21.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-08-19T11:28:22.169 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (a,c) after stopping b 2024-08-19T11:28:22.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-08-19T11:28:22.531 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (a,b) after stopping c 2024-08-19T11:28:22.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2024-08-19T11:28:22.866 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons a b 2024-08-19T11:28:22.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2024-08-19T11:28:23.107 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons b c 2024-08-19T11:28:23.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2024-08-19T11:28:23.420 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available () after stopping mons a b c 2024-08-19T11:28:23.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-08-19T11:28:23.902 INFO:tasks.workunit.client.0.smithi130.stderr:adding another mon that is not yet online will not break quorum 2024-08-19T11:28:23.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-08-19T11:28:24.228 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.a 2024-08-19T11:28:24.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-08-19T11:28:24.607 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.b 2024-08-19T11:28:24.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-08-19T11:28:24.972 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.c 2024-08-19T11:28:24.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2024-08-19T11:28:24.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:28:24.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:28:24.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:28:24.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:28:24.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:28:25.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:28:25.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2024-08-19T11:28:25.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-08-19T11:28:25.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=2 2024-08-19T11:28:25.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-08-19T11:28:25.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 2 ]] 2024-08-19T11:28:25.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-08-19T11:28:25.096 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-08-19T11:28:25.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1724066965 2024-08-19T11:28:25.097 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:25.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066905 -lt 1724066965 ]] 2024-08-19T11:28:25.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-08-19T11:28:25.099 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:28:25.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:28:25.338 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:25.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-08-19T11:28:25.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:25.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:28:25.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:28:25.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:25.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066905 -lt 1724066965 ]] 2024-08-19T11:28:25.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-08-19T11:28:25.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:28:28.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:28:28.603 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:28.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-08-19T11:28:28.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:28.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:28:28.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:28:28.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:28.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066908 -lt 1724066965 ]] 2024-08-19T11:28:28.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-08-19T11:28:28.640 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:28:28.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:28:28.856 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:28.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-08-19T11:28:28.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:28.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:28:28.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:28:28.885 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:28.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066908 -lt 1724066965 ]] 2024-08-19T11:28:28.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-08-19T11:28:28.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:28:29.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:28:29.121 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:29.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:29.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-08-19T11:28:29.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:28:29.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:28:29.155 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:29.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066909 -lt 1724066965 ]] 2024-08-19T11:28:29.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-08-19T11:28:29.157 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:28:29.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:28:29.380 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:29.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-08-19T11:28:29.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:29.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:28:29.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:28:29.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:29.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066909 -lt 1724066965 ]] 2024-08-19T11:28:29.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-08-19T11:28:29.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:28:29.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:28:29.642 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:29.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:29.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-08-19T11:28:29.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:28:29.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:28:29.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:29.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066909 -lt 1724066965 ]] 2024-08-19T11:28:29.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-08-19T11:28:29.675 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:28:32.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:28:32.912 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:32.912 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-08-19T11:28:32.913 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:32.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:28:32.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:28:32.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:32.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066912 -lt 1724066965 ]] 2024-08-19T11:28:32.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-08-19T11:28:32.941 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:28:33.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:28:33.178 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:33.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:33.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-08-19T11:28:33.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:28:33.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:28:33.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:33.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066913 -lt 1724066965 ]] 2024-08-19T11:28:33.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-08-19T11:28:33.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:28:33.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:28:33.440 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:33.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-08-19T11:28:33.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:33.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:28:33.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:28:33.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:33.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066913 -lt 1724066965 ]] 2024-08-19T11:28:33.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-08-19T11:28:33.471 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:28:33.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:28:33.705 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:33.706 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:33.706 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-08-19T11:28:33.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:28:33.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:28:33.741 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:33.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066913 -lt 1724066965 ]] 2024-08-19T11:28:33.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-08-19T11:28:33.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:28:33.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:28:33.975 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:33.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-08-19T11:28:33.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:34.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:28:34.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:28:34.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:34.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066914 -lt 1724066965 ]] 2024-08-19T11:28:34.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-08-19T11:28:34.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:28:40.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:28:40.506 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:40.506 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-08-19T11:28:40.506 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:09.173329Z","created":"2024-08-19T11:28:09.173329Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:28:40.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-08-19T11:28:40.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-08-19T11:28:40.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-08-19T11:28:40.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-08-19T11:28:40.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-08-19T11:28:40.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-08-19T11:28:40.829 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons a 2024-08-19T11:28:40.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-08-19T11:28:41.132 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (a,c) after stopping b 2024-08-19T11:28:41.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-08-19T11:28:44.334 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons c 2024-08-19T11:28:44.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-08-19T11:28:44.605 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-08-19T11:28:44.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-08-19T11:28:47.802 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: removing mon.a would break quorum 2024-08-19T11:28:47.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-08-19T11:28:48.122 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.b 2024-08-19T11:28:48.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-08-19T11:28:48.365 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: removing mon.c would break quorum 2024-08-19T11:28:48.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-08-19T11:28:48.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:28:48.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:28:48.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:28:48.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:28:48.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:28:48.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:28:48.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-08-19T11:28:48.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:28:48.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:28:48.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:28:48.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:28:48.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:28:48.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:28:48.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:28:48.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:28:48.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:28:48.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:28:48.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:28:48.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:28:48.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:28:48.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:28:48.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:28:48.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:28:48.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:28:48.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:28:48.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:28:48.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:28:48.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:28:48.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:28:48.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:28:48.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:48.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:48.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:28:48.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:28:48.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:28:48.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-08-19T11:28:48.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-08-19T11:28:48.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-08-19T11:28:48.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-08-19T11:28:48.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:28:48.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:28:48.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:28:48.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:28:48.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:28:48.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:28:48.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:28:48.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:28:48.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:28:48.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:28:48.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:28:48.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:28:48.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:28:48.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:28:48.503 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:28:48.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:28:48.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:28:48.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:28:48.505 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:28:48.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:28:48.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:28:48.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:28:48.508 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:28:48.508 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:48.508 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:48.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:28:48.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:28:48.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:28:48.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-08-19T11:28:48.510 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:28:48.510 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:48.510 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:48.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.48100 2024-08-19T11:28:48.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:28:48.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:28:48.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:28:48.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-08-19T11:28:48.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2024-08-19T11:28:48.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2024-08-19T11:28:48.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2024-08-19T11:28:48.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-08-19T11:28:48.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:28:48.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:28:48.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:28:48.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:28:48.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-08-19T11:28:48.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-08-19T11:28:48.984 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:28:48.984 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:28:48.985 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:28:48.986 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:28:48.986 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:48.986 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:48.986 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:28:48.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-08-19T11:28:49.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:28:49.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:28:49.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:49.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:28:49.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:28:49.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:28:49.179 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:28:49.179 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:28:49.179 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:28:49.180 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:49.180 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:49.180 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:49.180 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:28:49.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:49.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get fsid 2024-08-19T11:28:49.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:28:49.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:49.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:28:49.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:28:49.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:28:49.297 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:28:49.297 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:28:49.297 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:28:49.297 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:49.297 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:49.297 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:49.297 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:28:49.298 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:49.298 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get mon_host 2024-08-19T11:28:49.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-08-19T11:28:49.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:28:49.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:28:49.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-08-19T11:28:49.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:28:49.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-08-19T11:28:49.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-08-19T11:28:49.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:28:49.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:28:49.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:28:49.921 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:28:49.921 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:49.921 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:49.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:28:49.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-08-19T11:28:50.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:28:50.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-08-19T11:28:50.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:50.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-08-19T11:28:50.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:28:50.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:28:50.144 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-08-19T11:28:50.144 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-08-19T11:28:50.144 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-08-19T11:28:50.144 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:50.144 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:50.145 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:50.145 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.b.asok 2024-08-19T11:28:50.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:50.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.b.asok config get fsid 2024-08-19T11:28:50.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-08-19T11:28:50.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:50.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-08-19T11:28:50.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:28:50.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:28:50.256 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-08-19T11:28:50.256 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-08-19T11:28:50.256 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-08-19T11:28:50.257 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:50.257 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:50.257 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:50.257 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.b.asok 2024-08-19T11:28:50.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:50.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.b.asok config get mon_host 2024-08-19T11:28:50.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-08-19T11:28:50.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:28:50.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:28:50.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-08-19T11:28:50.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:28:50.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-08-19T11:28:50.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-08-19T11:28:50.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:28:50.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:28:50.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:28:50.800 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:28:50.800 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:50.801 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:50.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:28:50.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-08-19T11:28:51.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:28:51.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-08-19T11:28:51.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:51.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-08-19T11:28:51.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:28:51.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:28:51.030 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-08-19T11:28:51.030 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-08-19T11:28:51.030 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-08-19T11:28:51.030 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:51.030 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:51.030 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:51.031 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.c.asok 2024-08-19T11:28:51.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:51.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.c.asok config get fsid 2024-08-19T11:28:51.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-08-19T11:28:51.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:51.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-08-19T11:28:51.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:28:51.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:28:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-08-19T11:28:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-08-19T11:28:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-08-19T11:28:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.c.asok 2024-08-19T11:28:51.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:51.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.c.asok config get mon_host 2024-08-19T11:28:51.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-08-19T11:28:51.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:28:51.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:28:51.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-08-19T11:28:51.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:28:51.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-08-19T11:28:51.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-08-19T11:28:51.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:28:51.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:28:51.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:28:51.755 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:28:51.755 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:51.755 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:51.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:28:51.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-08-19T11:28:51.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:28:51.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-08-19T11:28:51.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:51.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-08-19T11:28:51.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:28:51.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:28:51.975 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-08-19T11:28:51.975 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-08-19T11:28:51.975 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-08-19T11:28:51.976 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:51.977 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:51.977 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:51.977 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.d.asok 2024-08-19T11:28:51.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:51.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.d.asok config get fsid 2024-08-19T11:28:52.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-08-19T11:28:52.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:28:52.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-08-19T11:28:52.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:28:52.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:28:52.093 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-08-19T11:28:52.094 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-08-19T11:28:52.094 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-08-19T11:28:52.094 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:28:52.094 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:28:52.094 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:28:52.095 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.d.asok 2024-08-19T11:28:52.095 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:28:52.095 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.d.asok config get mon_host 2024-08-19T11:28:52.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2024-08-19T11:28:52.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-08-19T11:28:52.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-08-19T11:28:52.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-08-19T11:28:52.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-08-19T11:28:52.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-08-19T11:28:52.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-08-19T11:28:52.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1724066992 2024-08-19T11:28:52.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:28:52.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066932 -lt 1724066992 ]] 2024-08-19T11:28:52.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:28:52.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:04.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:04.729 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:04.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:29:04.730 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:04.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-08-19T11:29:04.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-08-19T11:29:04.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-08-19T11:29:04.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-08-19T11:29:04.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-08-19T11:29:04.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2024-08-19T11:29:05.106 INFO:tasks.workunit.client.0.smithi130.stderr:invalid mon(s) specified: dne 2024-08-19T11:29:05.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-08-19T11:29:05.470 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (b,c,d) after stopping a 2024-08-19T11:29:05.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-08-19T11:29:05.830 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (a,c,d) after stopping b 2024-08-19T11:29:05.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-08-19T11:29:06.199 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (a,b,d) after stopping c 2024-08-19T11:29:06.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-08-19T11:29:06.521 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (a,b,c) after stopping d 2024-08-19T11:29:06.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2024-08-19T11:29:06.856 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons a b 2024-08-19T11:29:06.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2024-08-19T11:29:07.139 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons c d 2024-08-19T11:29:07.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-08-19T11:29:07.518 INFO:tasks.workunit.client.0.smithi130.stderr:adding another mon that is not yet online will not break quorum 2024-08-19T11:29:07.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-08-19T11:29:07.867 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.a 2024-08-19T11:29:07.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-08-19T11:29:08.235 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.b 2024-08-19T11:29:08.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-08-19T11:29:08.602 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.c 2024-08-19T11:29:08.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-08-19T11:29:08.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:29:08.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:29:08.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:29:08.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:29:08.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:29:08.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:29:08.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2024-08-19T11:29:08.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-08-19T11:29:08.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-08-19T11:29:08.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-08-19T11:29:08.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-08-19T11:29:08.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-08-19T11:29:08.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-08-19T11:29:08.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1724067008 2024-08-19T11:29:08.727 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:08.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066948 -lt 1724067008 ]] 2024-08-19T11:29:08.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:08.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:08.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:08.976 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:08.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:08.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:09.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:29:09.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:29:09.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:09.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066949 -lt 1724067008 ]] 2024-08-19T11:29:09.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:09.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:09.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:09.243 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:09.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:09.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:09.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:29:09.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:29:09.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:09.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066949 -lt 1724067008 ]] 2024-08-19T11:29:09.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:09.280 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:09.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:09.498 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:09.498 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:09.498 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:09.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:29:09.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:29:09.525 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:09.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066949 -lt 1724067008 ]] 2024-08-19T11:29:09.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:09.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:09.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:09.756 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:09.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:09.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:09.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:29:09.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:29:09.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:09.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066949 -lt 1724067008 ]] 2024-08-19T11:29:09.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:09.792 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:13.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:13.021 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:13.022 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:13.022 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:13.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:29:13.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:29:13.056 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:13.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066953 -lt 1724067008 ]] 2024-08-19T11:29:13.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:13.057 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:16.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:16.304 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:16.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:16.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:16.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:29:16.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:29:16.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:16.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066956 -lt 1724067008 ]] 2024-08-19T11:29:16.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:16.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:16.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:16.573 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:16.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:16.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:16.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:29:16.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:29:16.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:16.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066956 -lt 1724067008 ]] 2024-08-19T11:29:16.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:16.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:16.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:16.841 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:16.842 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:16.842 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:16.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:29:16.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:29:16.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:16.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066956 -lt 1724067008 ]] 2024-08-19T11:29:16.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:16.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:17.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:17.112 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:17.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:17.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:17.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:29:17.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:29:17.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:17.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066957 -lt 1724067008 ]] 2024-08-19T11:29:17.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:17.140 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:17.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:17.377 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:17.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:17.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:17.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:29:17.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:29:17.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:17.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066957 -lt 1724067008 ]] 2024-08-19T11:29:17.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:17.414 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:17.637 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:17.637 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:17.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:29:17.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:29:17.670 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:17.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066957 -lt 1724067008 ]] 2024-08-19T11:29:17.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:17.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:17.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:17.902 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:17.903 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:17.903 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:17.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:29:17.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:29:17.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:17.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066957 -lt 1724067008 ]] 2024-08-19T11:29:17.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:17.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:18.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:18.194 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:18.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:18.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:18.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:29:18.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:29:18.231 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:18.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066958 -lt 1724067008 ]] 2024-08-19T11:29:18.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:29:18.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:30.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:30.471 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:30.471 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:28:48.536927Z","created":"2024-08-19T11:28:48.536927Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:30.471 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:29:30.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-08-19T11:29:30.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-08-19T11:29:30.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-08-19T11:29:30.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-08-19T11:29:30.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-08-19T11:29:30.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-08-19T11:29:30.821 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (b,c,d) after stopping a 2024-08-19T11:29:30.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-08-19T11:29:31.151 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons b 2024-08-19T11:29:31.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-08-19T11:29:31.419 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons c 2024-08-19T11:29:31.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-08-19T11:29:31.720 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons d 2024-08-19T11:29:31.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-08-19T11:29:32.048 INFO:tasks.workunit.client.0.smithi130.stderr:adding another mon that is not yet online will not break quorum 2024-08-19T11:29:32.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-08-19T11:29:35.296 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.a 2024-08-19T11:29:35.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-08-19T11:29:35.660 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.b 2024-08-19T11:29:35.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-08-19T11:29:38.906 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.c 2024-08-19T11:29:38.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2024-08-19T11:29:39.264 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.d 2024-08-19T11:29:39.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-08-19T11:29:39.281 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:29:39.281 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:29:39.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:29:39.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:29:39.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:29:39.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:29:39.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-08-19T11:29:39.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:29:39.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:29:39.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:29:39.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:29:39.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:29:39.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:29:39.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:29:39.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:29:39.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:29:39.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:29:39.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:29:39.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:29:39.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:29:39.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:29:39.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:29:39.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:29:39.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:29:39.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:29:39.400 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:29:39.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:29:39.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:29:39.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:29:39.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:29:39.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:39.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:39.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:29:39.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:29:39.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:29:39.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-08-19T11:29:39.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-08-19T11:29:39.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-08-19T11:29:39.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-08-19T11:29:39.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:29:39.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:29:39.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:29:39.410 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:29:39.410 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:29:39.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:29:39.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:29:39.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:29:39.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:29:39.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:29:39.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:29:39.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:29:39.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:29:39.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:29:39.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:29:39.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:29:39.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:29:39.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:29:39.414 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:29:39.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:29:39.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:29:39.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:29:39.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:29:39.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:39.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:39.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:29:39.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:29:39.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:29:39.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-08-19T11:29:39.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:29:39.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:39.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:39.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.48100 2024-08-19T11:29:39.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:29:39.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:29:39.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:29:39.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-08-19T11:29:39.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2024-08-19T11:29:39.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2024-08-19T11:29:39.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2024-08-19T11:29:39.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-08-19T11:29:39.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:29:39.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:29:39.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:29:39.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:29:39.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-08-19T11:29:39.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-08-19T11:29:39.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:29:39.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:29:39.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:29:39.936 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:29:39.936 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:39.936 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:39.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:29:39.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-08-19T11:29:40.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:29:40.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:29:40.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:29:40.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:29:40.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:29:40.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:29:40.129 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:29:40.129 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:29:40.129 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:29:40.130 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:29:40.130 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:40.130 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:40.130 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:29:40.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:29:40.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get fsid 2024-08-19T11:29:40.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:29:40.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:29:40.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:29:40.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:29:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:29:40.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:29:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:29:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:29:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:29:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:40.252 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.a.asok 2024-08-19T11:29:40.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:29:40.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.a.asok config get mon_host 2024-08-19T11:29:40.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-08-19T11:29:40.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:29:40.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:29:40.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-08-19T11:29:40.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:29:40.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-08-19T11:29:40.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-08-19T11:29:40.821 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:29:40.821 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:29:40.821 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:29:40.822 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:29:40.822 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:40.822 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:40.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:29:40.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-08-19T11:29:41.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:29:41.039 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-08-19T11:29:41.039 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:29:41.040 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-08-19T11:29:41.040 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:29:41.041 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:29:41.041 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-08-19T11:29:41.041 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-08-19T11:29:41.041 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-08-19T11:29:41.041 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:29:41.041 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:41.041 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:41.041 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.b.asok 2024-08-19T11:29:41.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:29:41.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.b.asok config get fsid 2024-08-19T11:29:41.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-08-19T11:29:41.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:29:41.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-08-19T11:29:41.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:29:41.152 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-08-19T11:29:41.153 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-08-19T11:29:41.153 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-08-19T11:29:41.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:29:41.153 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:29:41.153 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:41.153 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:41.153 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.b.asok 2024-08-19T11:29:41.154 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:29:41.154 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.b.asok config get mon_host 2024-08-19T11:29:41.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-08-19T11:29:41.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:29:41.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:29:41.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-08-19T11:29:41.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:29:41.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-08-19T11:29:41.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-08-19T11:29:41.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:29:41.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:29:41.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:29:41.717 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:29:41.717 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:41.717 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:41.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:29:41.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-08-19T11:29:41.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:29:41.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-08-19T11:29:41.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:29:41.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-08-19T11:29:41.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:29:41.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:29:41.931 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-08-19T11:29:41.931 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-08-19T11:29:41.932 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-08-19T11:29:41.932 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:29:41.932 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:41.932 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:41.932 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.c.asok 2024-08-19T11:29:41.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:29:41.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.c.asok config get fsid 2024-08-19T11:29:42.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-08-19T11:29:42.048 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:29:42.048 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-08-19T11:29:42.048 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:29:42.048 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:29:42.048 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-08-19T11:29:42.048 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-08-19T11:29:42.049 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-08-19T11:29:42.049 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:29:42.049 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:42.049 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:42.049 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.c.asok 2024-08-19T11:29:42.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:29:42.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.c.asok config get mon_host 2024-08-19T11:29:42.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-08-19T11:29:42.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:29:42.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:29:42.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-08-19T11:29:42.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:29:42.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-08-19T11:29:42.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-08-19T11:29:42.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:29:42.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:29:42.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:29:42.655 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:29:42.655 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:42.655 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:42.655 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:29:42.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-08-19T11:29:42.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:29:42.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-08-19T11:29:42.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:29:42.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-08-19T11:29:42.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:29:42.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:29:42.866 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-08-19T11:29:42.867 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-08-19T11:29:42.867 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-08-19T11:29:42.867 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:29:42.867 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:42.867 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:42.867 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.d.asok 2024-08-19T11:29:42.867 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:29:42.867 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.d.asok config get fsid 2024-08-19T11:29:42.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-08-19T11:29:42.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:29:42.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-08-19T11:29:42.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:29:42.976 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-08-19T11:29:42.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:29:42.976 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-08-19T11:29:42.976 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-08-19T11:29:42.976 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:29:42.976 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:42.976 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:42.976 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.d.asok 2024-08-19T11:29:42.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:29:42.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.d.asok config get mon_host 2024-08-19T11:29:43.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2024-08-19T11:29:43.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-08-19T11:29:43.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:29:43.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=e 2024-08-19T11:29:43.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:29:43.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/e 2024-08-19T11:29:43.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2024-08-19T11:29:43.666 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:29:43.666 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:29:43.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:29:43.667 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:29:43.667 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:43.667 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:43.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48100/$cluster-$name.asok' 2024-08-19T11:29:43.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.48100/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2024-08-19T11:29:43.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:29:43.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e fsid 2024-08-19T11:29:43.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:29:43.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-08-19T11:29:43.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:29:43.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:29:43.898 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-08-19T11:29:43.898 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-08-19T11:29:43.898 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-08-19T11:29:43.899 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:29:43.899 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:43.899 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:43.899 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.e.asok 2024-08-19T11:29:43.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:29:43.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.e.asok config get fsid 2024-08-19T11:29:44.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e mon_host 2024-08-19T11:29:44.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:29:44.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-08-19T11:29:44.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:29:44.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:29:44.013 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-08-19T11:29:44.013 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-08-19T11:29:44.013 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-08-19T11:29:44.013 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:29:44.013 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:29:44.013 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:29:44.013 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48100/ceph-mon.e.asok 2024-08-19T11:29:44.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:29:44.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48100/ceph-mon.e.asok config get mon_host 2024-08-19T11:29:44.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2024-08-19T11:29:44.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-08-19T11:29:44.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=5 2024-08-19T11:29:44.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-08-19T11:29:44.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 5 ]] 2024-08-19T11:29:44.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-08-19T11:29:44.120 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-08-19T11:29:44.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1724067044 2024-08-19T11:29:44.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:29:44.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724066984 -lt 1724067044 ]] 2024-08-19T11:29:44.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-08-19T11:29:44.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:29:59.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:29:59.366 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:59.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:29:59.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 5' 2024-08-19T11:29:59.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-08-19T11:29:59.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-08-19T11:29:59.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-08-19T11:29:59.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-08-19T11:29:59.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-08-19T11:29:59.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2024-08-19T11:29:59.731 INFO:tasks.workunit.client.0.smithi130.stderr:invalid mon(s) specified: dne 2024-08-19T11:29:59.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-08-19T11:30:00.134 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (b,c,d,e) after stopping a 2024-08-19T11:30:00.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-08-19T11:30:03.382 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (a,c,d,e) after stopping b 2024-08-19T11:30:03.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-08-19T11:30:03.786 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (a,b,d,e) after stopping c 2024-08-19T11:30:03.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-08-19T11:30:04.133 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (a,b,c,e) after stopping d 2024-08-19T11:30:04.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-08-19T11:30:04.490 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (a,b,c,d) after stopping e 2024-08-19T11:30:04.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2024-08-19T11:30:04.821 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (c,d,e) after stopping a b 2024-08-19T11:30:04.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2024-08-19T11:30:05.201 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (a,b,e) after stopping c d 2024-08-19T11:30:05.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2024-08-19T11:30:05.545 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons a b c 2024-08-19T11:30:05.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-08-19T11:30:05.910 INFO:tasks.workunit.client.0.smithi130.stderr:adding another mon that is not yet online will not break quorum 2024-08-19T11:30:05.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-08-19T11:30:06.283 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.a 2024-08-19T11:30:06.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-08-19T11:30:06.627 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.b 2024-08-19T11:30:06.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-08-19T11:30:07.007 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.c 2024-08-19T11:30:07.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-08-19T11:30:07.345 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.d 2024-08-19T11:30:07.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-08-19T11:30:07.725 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.e 2024-08-19T11:30:07.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-08-19T11:30:07.743 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:30:07.743 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:30:07.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:30:07.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:30:07.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:30:07.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:30:07.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2024-08-19T11:30:07.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-08-19T11:30:07.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-08-19T11:30:07.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-08-19T11:30:07.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-08-19T11:30:07.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-08-19T11:30:07.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-08-19T11:30:07.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1724067067 2024-08-19T11:30:07.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:07.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067007 -lt 1724067067 ]] 2024-08-19T11:30:07.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:07.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:11.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:11.112 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:11.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:11.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:11.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:11.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:11.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:11.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067011 -lt 1724067067 ]] 2024-08-19T11:30:11.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:11.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:11.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:11.377 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:11.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:11.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:11.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:11.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:11.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:11.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067011 -lt 1724067067 ]] 2024-08-19T11:30:11.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:11.414 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:11.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:11.648 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:11.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:11.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:11.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:11.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:11.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:11.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067011 -lt 1724067067 ]] 2024-08-19T11:30:11.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:11.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:11.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:11.914 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:11.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:11.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:11.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:11.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:11.949 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:11.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067011 -lt 1724067067 ]] 2024-08-19T11:30:11.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:11.950 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:12.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:12.181 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:12.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:12.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:12.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:12.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:12.217 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:12.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067012 -lt 1724067067 ]] 2024-08-19T11:30:12.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:12.218 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:12.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:12.451 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:12.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:12.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:12.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:12.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:12.489 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:12.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067012 -lt 1724067067 ]] 2024-08-19T11:30:12.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:12.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:12.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:12.713 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:12.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:12.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:12.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:12.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:12.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:12.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067012 -lt 1724067067 ]] 2024-08-19T11:30:12.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:12.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:12.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:12.969 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:12.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:12.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:12.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:12.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:12.999 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:12.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067013 -lt 1724067067 ]] 2024-08-19T11:30:12.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:13.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:13.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:13.236 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:13.237 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:13.237 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:13.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:13.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:13.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:13.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067013 -lt 1724067067 ]] 2024-08-19T11:30:13.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:13.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:13.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:13.501 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:13.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:13.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:13.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:13.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:13.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067013 -lt 1724067067 ]] 2024-08-19T11:30:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:13.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:13.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:13.766 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:13.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:13.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:13.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:13.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:13.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:13.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067013 -lt 1724067067 ]] 2024-08-19T11:30:13.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:13.795 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:17.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:17.036 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:17.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:17.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:17.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:17.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:17.064 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:17.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067017 -lt 1724067067 ]] 2024-08-19T11:30:17.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:17.064 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:17.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:17.286 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:17.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:17.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:17.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:17.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:17.319 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:17.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067017 -lt 1724067067 ]] 2024-08-19T11:30:17.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:17.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:17.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:17.541 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:17.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:17.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:17.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:17.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:17.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:17.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067017 -lt 1724067067 ]] 2024-08-19T11:30:17.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-08-19T11:30:17.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:23.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:23.879 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:23.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:23.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-08-19T11:30:23.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-08-19T11:30:23.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-08-19T11:30:23.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-08-19T11:30:23.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-08-19T11:30:23.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-08-19T11:30:23.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-08-19T11:30:24.224 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (b,c,d,e) after stopping a 2024-08-19T11:30:24.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-08-19T11:30:24.621 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (c,d,e) after stopping b 2024-08-19T11:30:24.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-08-19T11:30:24.986 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (b,d,e) after stopping c 2024-08-19T11:30:25.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-08-19T11:30:25.354 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (b,c,e) after stopping d 2024-08-19T11:30:25.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-08-19T11:30:25.753 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (b,c,d) after stopping e 2024-08-19T11:30:25.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-08-19T11:30:26.174 INFO:tasks.workunit.client.0.smithi130.stderr:adding another mon that is not yet online will not break quorum 2024-08-19T11:30:26.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-08-19T11:30:29.415 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.a 2024-08-19T11:30:29.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-08-19T11:30:29.761 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.b 2024-08-19T11:30:29.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-08-19T11:30:30.085 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.c 2024-08-19T11:30:30.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-08-19T11:30:30.386 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.d 2024-08-19T11:30:30.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-08-19T11:30:30.724 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.e 2024-08-19T11:30:30.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2024-08-19T11:30:30.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:30:30.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:30:30.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:30:30.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:30:30.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:30:30.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:30:30.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2024-08-19T11:30:30.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-08-19T11:30:30.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-08-19T11:30:30.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-08-19T11:30:30.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-08-19T11:30:30.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-08-19T11:30:30.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-08-19T11:30:30.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1724067090 2024-08-19T11:30:30.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:30.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067030 -lt 1724067090 ]] 2024-08-19T11:30:30.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:30:30.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:31.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:31.086 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:31.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:31.087 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:30:31.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:31.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:31.120 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:31.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067031 -lt 1724067090 ]] 2024-08-19T11:30:31.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:30:31.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:31.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:31.347 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:31.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:31.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:30:31.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:31.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:31.382 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:31.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067031 -lt 1724067090 ]] 2024-08-19T11:30:31.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:30:31.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:31.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:31.618 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:31.619 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:30:31.619 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:31.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:31.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:31.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:31.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067031 -lt 1724067090 ]] 2024-08-19T11:30:31.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:30:31.655 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:37.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:37.890 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:37.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:37.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:30:37.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:37.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:37.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:37.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067037 -lt 1724067090 ]] 2024-08-19T11:30:37.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:30:37.926 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:38.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:38.148 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:38.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:30:38.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:38.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:38.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:38.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:38.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067038 -lt 1724067090 ]] 2024-08-19T11:30:38.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:30:38.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:38.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:38.411 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:38.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:30:38.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:38.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-08-19T11:30:38.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-08-19T11:30:38.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-08-19T11:30:38.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1724067038 -lt 1724067090 ]] 2024-08-19T11:30:38.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-08-19T11:30:38.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-08-19T11:30:50.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-08-19T11:30:50.674 INFO:tasks.workunit.client.0.smithi130.stderr:{"election_epoch":14,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:50.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-08-19T11:30:50.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1cd18d8f-7412-4aa8-b79e-b9e78d6d4d18","modified":"2024-08-19T11:29:39.444099Z","created":"2024-08-19T11:29:39.444099Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-08-19T11:30:50.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-08-19T11:30:50.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-08-19T11:30:50.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-08-19T11:30:50.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-08-19T11:30:50.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-08-19T11:30:50.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-08-19T11:30:50.977 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (b,c,d) after stopping a 2024-08-19T11:30:50.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-08-19T11:30:54.176 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons b 2024-08-19T11:30:54.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-08-19T11:30:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons c 2024-08-19T11:30:54.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-08-19T11:30:54.741 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons d 2024-08-19T11:30:54.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-08-19T11:30:57.976 INFO:tasks.workunit.client.0.smithi130.stderr:quorum should be preserved (b,c,d) after stopping e 2024-08-19T11:30:57.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-08-19T11:30:58.290 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-08-19T11:30:58.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-08-19T11:30:58.609 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.a 2024-08-19T11:30:58.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-08-19T11:31:01.799 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: removing mon.b would break quorum 2024-08-19T11:31:01.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-08-19T11:31:07.984 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: removing mon.c would break quorum 2024-08-19T11:31:07.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-08-19T11:31:11.173 INFO:tasks.workunit.client.0.smithi130.stderr:Error EBUSY: removing mon.d would break quorum 2024-08-19T11:31:11.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-08-19T11:31:11.477 INFO:tasks.workunit.client.0.smithi130.stderr:safe to remove mon.e 2024-08-19T11:31:11.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-08-19T11:31:11.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:31:11.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:31:11.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:31:11.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:31:11.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:31:11.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:31:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-08-19T11:31:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:31:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:31:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:31:11.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:31:11.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:31:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:31:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:31:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:31:11.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:31:11.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:31:11.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:31:11.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:31:11.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:31:11.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:31:11.616 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:31:11.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:31:11.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:31:11.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:31:11.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:31:11.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:31:11.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:31:11.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:31:11.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:31:11.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:31:11.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:31:11.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:31:11.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:31:11.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:31:11.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2331: main: code=0 2024-08-19T11:31:11.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: teardown td/ok-to-stop 0 2024-08-19T11:31:11.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-08-19T11:31:11.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-19T11:31:11.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-08-19T11:31:11.632 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:31:11.632 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:31:11.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:31:11.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:31:11.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:31:11.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:31:11.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:31:11.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:31:11.634 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:31:11.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:31:11.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:31:11.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:31:11.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:31:11.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:31:11.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:31:11.637 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:31:11.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:31:11.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-19T11:31:11.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-08-19T11:31:11.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:31:11.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:31:11.640 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48100 2024-08-19T11:31:11.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48100 2024-08-19T11:31:11.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:31:11.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:31:11.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: main: return 0 2024-08-19T11:31:11.642 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-19T11:31:11.642 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-19T11:31:11.652 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2024-08-19T11:31:11.652 DEBUG:teuthology.orchestra.run.smithi130:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2024-08-19T11:31:11.703 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-19T11:31:11.709 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-19T11:31:11.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-19T11:31:11.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-19T11:31:11.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export PYTHONWARNINGS=ignore 2024-08-19T11:31:11.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: PYTHONWARNINGS=ignore 2024-08-19T11:31:11.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: export CEPH_CONF=/dev/null 2024-08-19T11:31:11.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: CEPH_CONF=/dev/null 2024-08-19T11:31:11.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: unset CEPH_ARGS 2024-08-19T11:31:11.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: local code 2024-08-19T11:31:11.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: run td/rados-striper 2024-08-19T11:31:11.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2024-08-19T11:31:11.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2024-08-19T11:31:11.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2024-08-19T11:31:11.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2024-08-19T11:31:11.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2024-08-19T11:31:11.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2024-08-19T11:31:11.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=0b445ccb-77f4-44d1-b6ac-2ae92b97084c --auth-supported=none ' 2024-08-19T11:31:11.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2024-08-19T11:31:11.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2024-08-19T11:31:11.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/rados-striper 2024-08-19T11:31:11.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/rados-striper 2024-08-19T11:31:11.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-08-19T11:31:11.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:31:11.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-08-19T11:31:11.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:31:11.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:31:11.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:31:11.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:31:11.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:31:11.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:31:11.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:31:11.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:31:11.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:31:11.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:31:11.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:31:11.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:31:11.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:31:11.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:31:11.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:31:11.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:31:11.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:31:11.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:31:11.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-08-19T11:31:11.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:31:11.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:31:11.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65176 2024-08-19T11:31:11.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.65176 2024-08-19T11:31:11.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:31:11.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:31:11.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/rados-striper 2024-08-19T11:31:11.720 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:31:11.720 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:31:11.720 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65176 2024-08-19T11:31:11.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.65176 2024-08-19T11:31:11.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:31:11.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-19T11:31:11.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-19T11:31:11.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:31:11.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2024-08-19T11:31:11.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2024-08-19T11:31:11.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/rados-striper 2024-08-19T11:31:11.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:31:11.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:31:11.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:31:11.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/rados-striper/a 2024-08-19T11:31:11.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2024-08-19T11:31:12.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:31:12.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:31:12.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:31:12.185 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:31:12.185 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:31:12.185 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65176 2024-08-19T11:31:12.185 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65176/$cluster-$name.asok' 2024-08-19T11:31:12.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.65176/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:31:12.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:31:12.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:31:12.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:31:12.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:31:12.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:31:12.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:31:12.390 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:31:12.390 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:31:12.390 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:31:12.390 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:31:12.390 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:31:12.390 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65176 2024-08-19T11:31:12.394 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65176/ceph-mon.a.asok 2024-08-19T11:31:12.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:31:12.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.65176/ceph-mon.a.asok config get fsid 2024-08-19T11:31:12.506 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:31:12.506 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:31:12.506 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:31:12.506 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:31:12.507 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:31:12.507 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:31:12.507 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:31:12.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:31:12.507 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:31:12.507 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:31:12.507 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65176 2024-08-19T11:31:12.507 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.65176/ceph-mon.a.asok 2024-08-19T11:31:12.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:31:12.508 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.65176/ceph-mon.a.asok config get mon_host 2024-08-19T11:31:12.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2024-08-19T11:31:12.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-08-19T11:31:12.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:31:12.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:31:12.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:31:12.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/0 2024-08-19T11:31:12.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0b445ccb-77f4-44d1-b6ac-2ae92b97084c --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-08-19T11:31:12.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:31:12.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:31:12.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:31:12.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2024-08-19T11:31:12.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2024-08-19T11:31:12.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:31:12.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:31:12.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-08-19T11:31:12.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:31:12.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:31:12.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:31:12.611 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:31:12.611 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:31:12.611 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65176 2024-08-19T11:31:12.612 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65176/$cluster-$name.asok' 2024-08-19T11:31:12.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.65176/$cluster-$name.asok' 2024-08-19T11:31:12.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:31:12.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:31:12.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:31:12.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-08-19T11:31:12.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-08-19T11:31:12.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:31:12.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:31:12.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:31:12.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:31:12.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:31:12.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:31:12.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/0 2024-08-19T11:31:12.613 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:31:12.614 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 3fd2959c-0ab1-4c24-be70-a4ddd4c3b13a 2024-08-19T11:31:12.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3fd2959c-0ab1-4c24-be70-a4ddd4c3b13a 2024-08-19T11:31:12.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3fd2959c-0ab1-4c24-be70-a4ddd4c3b13a' 2024-08-19T11:31:12.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:31:12.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAALcNmqciUJRAAnZupojpTHbBYjXGzS+oqXQ== 2024-08-19T11:31:12.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAALcNmqciUJRAAnZupojpTHbBYjXGzS+oqXQ=="}' 2024-08-19T11:31:12.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3fd2959c-0ab1-4c24-be70-a4ddd4c3b13a -i td/rados-striper/0/new.json 2024-08-19T11:31:13.548 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:31:13.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/0/new.json 2024-08-19T11:31:13.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0b445ccb-77f4-44d1-b6ac-2ae92b97084c --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.65176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAALcNmqciUJRAAnZupojpTHbBYjXGzS+oqXQ== --osd-uuid 3fd2959c-0ab1-4c24-be70-a4ddd4c3b13a 2024-08-19T11:31:13.602 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:13.602+0000 7fa1e75467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:31:13.616 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:13.618+0000 7fa1e75467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:31:13.618 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:13.618+0000 7fa1e75467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:31:13.618 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:13.618+0000 7fa1e75467c0 -1 bdev(0x5612809f3c00 td/rados-striper/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:31:13.618 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:13.618+0000 7fa1e75467c0 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2024-08-19T11:31:19.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/0/keyring 2024-08-19T11:31:19.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:31:19.838 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:31:19.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:31:19.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:31:20.086 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:31:20.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:31:20.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:31:20.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0b445ccb-77f4-44d1-b6ac-2ae92b97084c --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.65176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:31:20.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:31:20.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:31:20.117 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:20.118+0000 7f46f117f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:31:20.119 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:20.122+0000 7f46f117f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:31:20.120 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:20.122+0000 7f46f117f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:31:20.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:31:20.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:31:20.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:31:20.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:31:20.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:31:20.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:20.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:31:20.317 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:31:20.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:20.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:20.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:21.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:21.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:21.550 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:31:21.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:31:21.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:21.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:21.653 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:21.654+0000 7f46f117f7c0 -1 Falling back to public interface 2024-08-19T11:31:21.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:22.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:22.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:22.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:31:22.778 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:31:22.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:22.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:23.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:24.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:24.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:24.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:31:24.012 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:31:24.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:24.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:24.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:24.505 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:24.506+0000 7f46f117f7c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:31:25.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:25.250 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:31:25.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:25.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:31:25.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:25.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:25.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:26.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:26.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:26.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:31:26.629 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:31:26.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:26.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:26.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:27.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:27.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:27.870 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:31:27.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:31:27.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:27.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:28.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:29.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:29.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:29.108 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:31:29.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:31:29.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:29.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:29.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:30.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:30.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:30.343 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:31:30.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:31:30.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:30.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:30.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:31.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:31.573 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:31:31.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:31.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:31:31.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:31.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:31.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:32.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:32.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:32.807 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:31:32.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:31:32.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:32.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:33.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:34.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:34.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:34.036 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:31:34.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:31:34.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:34.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:34.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:35.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:35.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:35.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:31:35.270 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:31:35.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:35.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:35.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:36.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:36.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:36.508 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:31:36.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:31:36.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:36.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:36.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:37.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:37.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:37.745 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:31:37.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:31:37.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:37.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:37.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:38.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:38.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:38.984 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:31:38.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:31:38.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:38.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:39.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:40.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:40.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:40.224 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:31:40.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:31:40.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:40.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:40.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:41.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:41.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:41.469 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:31:41.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:31:41.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:41.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:41.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:42.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:42.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:42.710 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:31:42.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:31:42.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:42.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:42.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:43.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:43.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:43.948 INFO:tasks.workunit.client.0.smithi130.stdout:19 2024-08-19T11:31:43.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T11:31:43.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:43.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:44.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:45.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:45.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:45.187 INFO:tasks.workunit.client.0.smithi130.stdout:20 2024-08-19T11:31:45.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-08-19T11:31:45.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:45.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:45.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:46.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:46.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:46.429 INFO:tasks.workunit.client.0.smithi130.stdout:21 2024-08-19T11:31:46.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-08-19T11:31:46.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:46.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:46.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:47.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:47.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:47.667 INFO:tasks.workunit.client.0.smithi130.stdout:22 2024-08-19T11:31:47.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-08-19T11:31:47.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:47.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:31:47.984 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/524802278,v1:127.0.0.1:6801/524802278] [v2:127.0.0.1:6802/524802278,v1:127.0.0.1:6803/524802278] exists,up 3fd2959c-0ab1-4c24-be70-a4ddd4c3b13a 2024-08-19T11:31:47.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:31:47.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:31:47.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:31:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2024-08-19T11:31:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-08-19T11:31:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:31:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T11:31:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:31:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/1 2024-08-19T11:31:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0b445ccb-77f4-44d1-b6ac-2ae92b97084c --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-08-19T11:31:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:31:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:31:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:31:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2024-08-19T11:31:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2024-08-19T11:31:47.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:31:47.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:31:47.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-08-19T11:31:47.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:31:47.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:31:47.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:31:47.988 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:31:47.988 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:31:47.988 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65176 2024-08-19T11:31:47.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65176/$cluster-$name.asok' 2024-08-19T11:31:47.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.65176/$cluster-$name.asok' 2024-08-19T11:31:47.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:31:47.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:31:47.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:31:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-08-19T11:31:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-08-19T11:31:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:31:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:31:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:31:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:31:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:31:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:31:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/1 2024-08-19T11:31:47.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:31:47.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fe4ab56b-22d3-4fcc-a019-595edaa3db94 2024-08-19T11:31:47.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fe4ab56b-22d3-4fcc-a019-595edaa3db94' 2024-08-19T11:31:47.990 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 fe4ab56b-22d3-4fcc-a019-595edaa3db94 2024-08-19T11:31:47.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:31:48.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAkLcNm6bmiABAAXGxrFkBTQz/TN7BlbizuEg== 2024-08-19T11:31:48.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAkLcNm6bmiABAAXGxrFkBTQz/TN7BlbizuEg=="}' 2024-08-19T11:31:48.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fe4ab56b-22d3-4fcc-a019-595edaa3db94 -i td/rados-striper/1/new.json 2024-08-19T11:31:48.399 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:31:48.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/1/new.json 2024-08-19T11:31:48.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0b445ccb-77f4-44d1-b6ac-2ae92b97084c --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.65176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkLcNm6bmiABAAXGxrFkBTQz/TN7BlbizuEg== --osd-uuid fe4ab56b-22d3-4fcc-a019-595edaa3db94 2024-08-19T11:31:48.452 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:48.453+0000 7f25f46977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:31:48.494 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:48.493+0000 7f25f46977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:31:48.495 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:48.497+0000 7f25f46977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:31:48.496 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:48.497+0000 7f25f46977c0 -1 bdev(0x5635a6e6bc00 td/rados-striper/1/block) open stat got: (1) Operation not permitted 2024-08-19T11:31:48.496 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:48.497+0000 7f25f46977c0 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2024-08-19T11:31:55.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/1/keyring 2024-08-19T11:31:55.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:31:55.258 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-08-19T11:31:55.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T11:31:55.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:31:55.507 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-08-19T11:31:55.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T11:31:55.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0b445ccb-77f4-44d1-b6ac-2ae92b97084c --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.65176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:31:55.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:31:55.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:31:55.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:31:55.532 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:55.533+0000 7fe64d2537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:31:55.535 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:55.537+0000 7fe64d2537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:31:55.536 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:55.537+0000 7fe64d2537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:31:55.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T11:31:55.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:31:55.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T11:31:55.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:31:55.740 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:31:55.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:31:55.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:55.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:31:55.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:55.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:31:55.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:56.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:56.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:56.966 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:31:56.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:31:56.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:56.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:31:57.049 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:57.049+0000 7fe64d2537c0 -1 Falling back to public interface 2024-08-19T11:31:57.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:58.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:58.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:58.193 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:31:58.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:31:58.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:58.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:31:58.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:59.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:31:59.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:31:59.424 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:31:59.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:31:59.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:31:59.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:31:59.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:31:59.943 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:31:59.941+0000 7fe64d2537c0 -1 osd.1 0 log_to_monitors true 2024-08-19T11:32:00.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:00.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:00.667 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:32:00.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:32:00.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:00.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:00.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:01.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:01.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:01.928 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:32:01.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:32:01.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:01.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:02.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:03.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:03.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:03.168 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:32:03.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:32:03.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:03.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:03.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:04.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:04.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:04.405 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:32:04.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:32:04.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:04.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:04.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:05.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:05.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:05.649 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:32:05.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:32:05.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:05.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:05.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:06.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:06.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:06.887 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:32:06.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:32:06.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:06.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:07.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:08.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:08.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:08.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:32:08.123 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:32:08.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:08.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:08.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:09.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:09.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:09.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:32:09.361 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:32:09.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:09.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:09.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:10.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:10.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:10.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:32:10.599 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:32:10.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:10.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:10.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:11.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:11.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:11.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:32:11.840 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:32:11.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:11.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:12.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:13.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:13.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:13.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:32:13.078 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:32:13.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:13.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:13.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:14.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:14.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:14.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:32:14.315 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:32:14.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:14.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:14.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:15.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:15.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:15.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:32:15.549 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:32:15.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:15.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:15.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:16.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:16.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:16.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:32:16.785 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:32:16.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:16.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:17.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:18.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:18.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:18.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:32:18.028 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:32:18.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:18.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:18.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:19.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:19.267 INFO:tasks.workunit.client.0.smithi130.stdout:19 2024-08-19T11:32:19.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:19.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T11:32:19.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:19.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:19.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:20.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:20.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-08-19T11:32:20.507 INFO:tasks.workunit.client.0.smithi130.stdout:20 2024-08-19T11:32:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:20.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:21.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:21.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:21.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-08-19T11:32:21.745 INFO:tasks.workunit.client.0.smithi130.stdout:21 2024-08-19T11:32:21.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:21.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:21.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:22.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:22.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:22.986 INFO:tasks.workunit.client.0.smithi130.stdout:22 2024-08-19T11:32:22.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-08-19T11:32:22.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:22.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:32:23.352 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/54275732,v1:127.0.0.1:6809/54275732] [v2:127.0.0.1:6810/54275732,v1:127.0.0.1:6811/54275732] exists,up fe4ab56b-22d3-4fcc-a019-595edaa3db94 2024-08-19T11:32:23.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:32:23.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:32:23.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:32:23.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2024-08-19T11:32:23.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-08-19T11:32:23.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:32:23.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-19T11:32:23.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:32:23.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/2 2024-08-19T11:32:23.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0b445ccb-77f4-44d1-b6ac-2ae92b97084c --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-08-19T11:32:23.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:32:23.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:32:23.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:32:23.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2024-08-19T11:32:23.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2024-08-19T11:32:23.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:32:23.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:32:23.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-08-19T11:32:23.359 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:32:23.360 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:32:23.360 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:32:23.360 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:32:23.360 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:32:23.360 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65176 2024-08-19T11:32:23.360 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.65176/$cluster-$name.asok' 2024-08-19T11:32:23.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.65176/$cluster-$name.asok' 2024-08-19T11:32:23.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:32:23.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:32:23.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:32:23.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-08-19T11:32:23.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-08-19T11:32:23.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:32:23.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:32:23.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:32:23.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:32:23.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:32:23.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:32:23.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/2 2024-08-19T11:32:23.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:32:23.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=36e97444-4497-4def-bc33-7f239b3fadf6 2024-08-19T11:32:23.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 36e97444-4497-4def-bc33-7f239b3fadf6' 2024-08-19T11:32:23.364 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 36e97444-4497-4def-bc33-7f239b3fadf6 2024-08-19T11:32:23.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:32:23.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHLcNm3pWqFhAAhxtxt21D4+H1S9iCumcxEg== 2024-08-19T11:32:23.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHLcNm3pWqFhAAhxtxt21D4+H1S9iCumcxEg=="}' 2024-08-19T11:32:23.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 36e97444-4497-4def-bc33-7f239b3fadf6 -i td/rados-striper/2/new.json 2024-08-19T11:32:23.787 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:32:23.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/2/new.json 2024-08-19T11:32:23.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0b445ccb-77f4-44d1-b6ac-2ae92b97084c --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.65176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHLcNm3pWqFhAAhxtxt21D4+H1S9iCumcxEg== --osd-uuid 36e97444-4497-4def-bc33-7f239b3fadf6 2024-08-19T11:32:23.838 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:32:23.836+0000 7efe518c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:32:23.914 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:32:23.912+0000 7efe518c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:32:23.915 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:32:23.916+0000 7efe518c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:32:23.916 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:32:23.916+0000 7efe518c77c0 -1 bdev(0x55c15ed55c00 td/rados-striper/2/block) open stat got: (1) Operation not permitted 2024-08-19T11:32:23.916 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:32:23.916+0000 7efe518c77c0 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2024-08-19T11:32:30.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/2/keyring 2024-08-19T11:32:30.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:32:30.371 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-08-19T11:32:30.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-19T11:32:30.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:32:30.620 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-08-19T11:32:30.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-19T11:32:30.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0b445ccb-77f4-44d1-b6ac-2ae92b97084c --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.65176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:32:30.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:32:30.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:32:30.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:32:30.652 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:32:30.652+0000 7f99173407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:32:30.654 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:32:30.652+0000 7f99173407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:32:30.656 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:32:30.656+0000 7f99173407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:32:30.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-19T11:32:30.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:32:30.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-19T11:32:30.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:32:30.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:32:30.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:30.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:32:30.842 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:32:30.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:30.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:31.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:31.954 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:32:31.952+0000 7f99173407c0 -1 Falling back to public interface 2024-08-19T11:32:32.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:32.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:32.072 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:32:32.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:32:32.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:32.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:32.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:33.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:33.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:33.310 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:32:33.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:32:33.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:33.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:33.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:34.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:34.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:34.540 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:32:34.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:32:34.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:34.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:34.623 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:32:34.620+0000 7f99173407c0 -1 osd.2 0 log_to_monitors true 2024-08-19T11:32:34.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:35.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:35.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:35.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:32:35.898 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:32:35.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:35.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:36.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:37.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:37.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:37.126 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:32:37.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:32:37.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:37.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:37.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:38.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:38.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:38.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:32:38.364 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:32:38.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:38.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:38.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:39.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:39.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:39.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:32:39.593 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:32:39.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:39.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:39.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:40.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:40.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:40.831 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:32:40.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:32:40.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:40.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:41.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:42.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:42.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:42.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:32:42.067 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:32:42.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:42.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:42.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:43.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:43.309 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:32:43.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:43.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:32:43.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:43.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:43.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:44.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:44.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:44.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:32:44.538 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:32:44.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:44.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:44.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:45.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:45.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:45.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:32:45.774 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:32:45.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:45.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:46.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:47.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:47.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:47.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:32:47.011 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:32:47.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:47.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:47.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:48.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:48.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:48.246 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:32:48.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:32:48.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:48.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:48.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:49.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:49.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:49.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:32:49.484 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:32:49.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:49.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:49.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:50.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:50.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:50.721 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:32:50.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:32:50.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:50.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:50.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:51.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:51.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:51.946 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:32:51.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:32:51.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:51.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:52.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:53.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:53.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:53.183 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:32:53.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:32:53.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:53.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:53.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:54.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:54.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:54.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T11:32:54.424 INFO:tasks.workunit.client.0.smithi130.stdout:19 2024-08-19T11:32:54.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:54.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:54.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:55.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:55.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:55.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-08-19T11:32:55.661 INFO:tasks.workunit.client.0.smithi130.stdout:20 2024-08-19T11:32:55.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:55.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:55.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:56.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:56.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:56.895 INFO:tasks.workunit.client.0.smithi130.stdout:21 2024-08-19T11:32:56.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-08-19T11:32:56.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:56.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:57.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:32:58.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:32:58.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:32:58.130 INFO:tasks.workunit.client.0.smithi130.stdout:22 2024-08-19T11:32:58.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-08-19T11:32:58.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:32:58.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-19T11:32:58.364 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/1404131037,v1:127.0.0.1:6817/1404131037] [v2:127.0.0.1:6818/1404131037,v1:127.0.0.1:6819/1404131037] exists,up 36e97444-4497-4def-bc33-7f239b3fadf6 2024-08-19T11:32:58.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:32:58.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:32:58.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:32:58.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2024-08-19T11:32:58.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:32:58.677 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:32:58.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:32:58.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:32:59.125 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:32:59.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:33:00.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:33:02.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2024-08-19T11:33:02.730 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records in 2024-08-19T11:33:02.730 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records out 2024-08-19T11:33:02.731 INFO:tasks.workunit.client.0.smithi130.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 0.000224286 s, 5.5 MB/s 2024-08-19T11:33:02.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2024-08-19T11:33:03.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2024-08-19T11:33:03.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2024-08-19T11:33:03.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2024-08-19T11:33:03.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2024-08-19T11:33:03.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2024-08-19T11:33:03.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2024-08-19T11:33:03.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2024-08-19T11:33:03.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2024-08-19T11:33:03.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2024-08-19T11:33:03.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2024-08-19T11:33:03.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2024-08-19T11:33:03.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2024-08-19T11:33:03.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2024-08-19T11:33:03.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2024-08-19T11:33:04.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2024-08-19T11:33:04.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2024-08-19T11:33:04.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2024-08-19T11:33:04.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2024-08-19T11:33:04.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2024-08-19T11:33:04.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2024-08-19T11:33:04.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2024-08-19T11:33:04.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2024-08-19T11:33:04.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2024-08-19T11:33:04.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-08-19T11:33:04.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2024-08-19T11:33:04.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2024-08-19T11:33:04.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2024-08-19T11:33:04.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2024-08-19T11:33:04.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2024-08-19T11:33:04.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2024-08-19T11:33:04.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2024-08-19T11:33:04.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2024-08-19T11:33:04.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2024-08-19T11:33:04.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2024-08-19T11:33:04.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2024-08-19T11:33:04.838 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2024-08-19T11:33:04.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2024-08-19T11:33:04.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2024-08-19T11:33:04.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: local dir=td/rados-striper 2024-08-19T11:33:04.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: shift 2024-08-19T11:33:04.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: local 'expected=No data available' 2024-08-19T11:33:04.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: shift 2024-08-19T11:33:04.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: local success 2024-08-19T11:33:04.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2024-08-19T11:33:04.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: expect_failure: success=false 2024-08-19T11:33:04.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: false 2024-08-19T11:33:04.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-08-19T11:33:04.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1995: expect_failure: return 0 2024-08-19T11:33:04.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-08-19T11:33:04.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: local dir=td/rados-striper 2024-08-19T11:33:04.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: shift 2024-08-19T11:33:04.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: local 'expected=No data available' 2024-08-19T11:33:04.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: shift 2024-08-19T11:33:04.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: local success 2024-08-19T11:33:04.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-08-19T11:33:04.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: expect_failure: success=false 2024-08-19T11:33:04.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: false 2024-08-19T11:33:04.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-08-19T11:33:04.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1995: expect_failure: return 0 2024-08-19T11:33:04.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2024-08-19T11:33:05.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2024-08-19T11:33:05.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: local dir=td/rados-striper 2024-08-19T11:33:05.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: shift 2024-08-19T11:33:05.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: local 'expected=No such file or directory' 2024-08-19T11:33:05.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: shift 2024-08-19T11:33:05.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: local success 2024-08-19T11:33:05.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: expect_failure: rados --pool rbd --striper stat toyfile 2024-08-19T11:33:05.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: expect_failure: success=false 2024-08-19T11:33:05.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: false 2024-08-19T11:33:05.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-08-19T11:33:05.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1995: expect_failure: return 0 2024-08-19T11:33:05.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2024-08-19T11:33:05.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: local dir=td/rados-striper 2024-08-19T11:33:05.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: shift 2024-08-19T11:33:05.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: local 'expected=No such file or directory' 2024-08-19T11:33:05.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: shift 2024-08-19T11:33:05.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: local success 2024-08-19T11:33:05.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2024-08-19T11:33:05.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: expect_failure: success=false 2024-08-19T11:33:05.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: false 2024-08-19T11:33:05.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-08-19T11:33:05.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1995: expect_failure: return 0 2024-08-19T11:33:05.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2024-08-19T11:33:05.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-08-19T11:33:05.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:33:05.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-08-19T11:33:05.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:33:05.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:33:05.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:33:05.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:33:05.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:33:05.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:33:05.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:33:05.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:33:05.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:33:05.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:33:05.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:33:05.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:33:05.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:33:05.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:33:05.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:33:05.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:33:05.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:33:05.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:33:05.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-08-19T11:33:05.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:33:05.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:05.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65176 2024-08-19T11:33:05.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.65176 2024-08-19T11:33:05.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:33:05.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:33:05.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2331: main: code=0 2024-08-19T11:33:05.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: teardown td/rados-striper 0 2024-08-19T11:33:05.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-08-19T11:33:05.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-19T11:33:05.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-08-19T11:33:05.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:33:05.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:33:05.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:33:05.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:33:05.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:33:05.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:33:05.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:33:05.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:33:05.319 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:33:05.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:33:05.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:33:05.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:33:05.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:33:05.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:33:05.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:33:05.322 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:33:05.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:33:05.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-19T11:33:05.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-08-19T11:33:05.325 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:33:05.325 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:05.325 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.65176 2024-08-19T11:33:05.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.65176 2024-08-19T11:33:05.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:33:05.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:33:05.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: main: return 0 2024-08-19T11:33:05.328 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-19T11:33:05.328 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-19T11:33:05.386 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2024-08-19T11:33:05.386 DEBUG:teuthology.orchestra.run.smithi130:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2024-08-19T11:33:05.445 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-19T11:33:05.450 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-19T11:33:05.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: run_tests: export .:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-19T11:33:05.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2345: export: `.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin': not a valid identifier 2024-08-19T11:33:05.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: export CEPH_MON=127.0.0.1:7109 2024-08-19T11:33:05.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: CEPH_MON=127.0.0.1:7109 2024-08-19T11:33:05.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: export CEPH_ARGS 2024-08-19T11:33:05.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: run_tests: uuidgen 2024-08-19T11:33:05.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: run_tests: CEPH_ARGS+=' --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none ' 2024-08-19T11:33:05.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2024-08-19T11:33:05.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: run_tests: export CEPH_CONF=/dev/null 2024-08-19T11:33:05.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: run_tests: CEPH_CONF=/dev/null 2024-08-19T11:33:05.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: run_tests: set 2024-08-19T11:33:05.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2024-08-19T11:33:05.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: run_tests: local 'funcs=test_activate_osd 2024-08-19T11:33:05.455 INFO:tasks.workunit.client.0.smithi130.stderr:test_activate_osd_after_mark_down 2024-08-19T11:33:05.455 INFO:tasks.workunit.client.0.smithi130.stderr:test_activate_osd_skip_benchmark 2024-08-19T11:33:05.455 INFO:tasks.workunit.client.0.smithi130.stderr:test_destroy_osd 2024-08-19T11:33:05.456 INFO:tasks.workunit.client.0.smithi130.stderr:test_display_logs 2024-08-19T11:33:05.456 INFO:tasks.workunit.client.0.smithi130.stderr:test_erasure_code_plugin_exists 2024-08-19T11:33:05.456 INFO:tasks.workunit.client.0.smithi130.stderr:test_expect_failure 2024-08-19T11:33:05.456 INFO:tasks.workunit.client.0.smithi130.stderr:test_flush_pg_stats 2024-08-19T11:33:05.456 INFO:tasks.workunit.client.0.smithi130.stderr:test_get_config 2024-08-19T11:33:05.456 INFO:tasks.workunit.client.0.smithi130.stderr:test_get_is_making_recovery_progress 2024-08-19T11:33:05.456 INFO:tasks.workunit.client.0.smithi130.stderr:test_get_last_scrub_stamp 2024-08-19T11:33:05.456 INFO:tasks.workunit.client.0.smithi130.stderr:test_get_not_primary 2024-08-19T11:33:05.456 INFO:tasks.workunit.client.0.smithi130.stderr:test_get_num_active_clean 2024-08-19T11:33:05.456 INFO:tasks.workunit.client.0.smithi130.stderr:test_get_num_active_or_peered 2024-08-19T11:33:05.456 INFO:tasks.workunit.client.0.smithi130.stderr:test_get_num_pgs 2024-08-19T11:33:05.456 INFO:tasks.workunit.client.0.smithi130.stderr:test_get_op_scheduler 2024-08-19T11:33:05.456 INFO:tasks.workunit.client.0.smithi130.stderr:test_get_osd_id_used_by_pgs 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_get_osds 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_get_pg 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_get_primary 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_get_timeout_delays 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_is_clean 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_kill_daemon 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_kill_daemons 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_objectstore_tool 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_pg_schedule_scrub 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_pg_scrub 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_repair 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_run_in_background 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_run_mon 2024-08-19T11:33:05.457 INFO:tasks.workunit.client.0.smithi130.stderr:test_run_osd 2024-08-19T11:33:05.458 INFO:tasks.workunit.client.0.smithi130.stderr:test_set_config 2024-08-19T11:33:05.458 INFO:tasks.workunit.client.0.smithi130.stderr:test_setup 2024-08-19T11:33:05.458 INFO:tasks.workunit.client.0.smithi130.stderr:test_teardown 2024-08-19T11:33:05.458 INFO:tasks.workunit.client.0.smithi130.stderr:test_wait_background 2024-08-19T11:33:05.458 INFO:tasks.workunit.client.0.smithi130.stderr:test_wait_for_clean 2024-08-19T11:33:05.458 INFO:tasks.workunit.client.0.smithi130.stderr:test_wait_for_health_ok 2024-08-19T11:33:05.458 INFO:tasks.workunit.client.0.smithi130.stderr:test_wait_for_osd 2024-08-19T11:33:05.458 INFO:tasks.workunit.client.0.smithi130.stderr:test_wait_for_peered 2024-08-19T11:33:05.458 INFO:tasks.workunit.client.0.smithi130.stderr:test_wait_for_scrub 2024-08-19T11:33:05.458 INFO:tasks.workunit.client.0.smithi130.stderr:test_wait_osd_id_used_by_pgs' 2024-08-19T11:33:05.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: run_tests: local dir=td/ceph-helpers 2024-08-19T11:33:05.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:33:05.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_activate_osd td/ceph-helpers 2024-08-19T11:33:05.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local dir=td/ceph-helpers 2024-08-19T11:33:05.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:888: test_activate_osd: setup td/ceph-helpers 2024-08-19T11:33:05.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:33:05.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:33:05.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:33:05.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:33:05.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:33:05.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:33:05.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:33:05.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:33:05.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:33:05.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:33:05.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:33:05.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:33:05.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:33:05.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:33:05.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:33:05.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:33:05.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:33:05.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:33:05.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:33:05.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:33:05.462 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:33:05.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:33:05.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:33:05.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:33:05.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:33:05.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:05.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:05.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:33:05.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:33:05.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:33:05.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:33:05.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:33:05.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:05.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:05.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:33:05.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:33:05.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-19T11:33:05.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-19T11:33:05.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:33:05.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:33:05.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:890: test_activate_osd: run_mon td/ceph-helpers a 2024-08-19T11:33:05.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:33:05.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:33:05.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:33:05.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:33:05.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:33:05.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-08-19T11:33:05.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:33:05.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:33:05.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:33:05.971 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:33:05.971 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:05.971 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:05.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:33:05.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:33:06.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:33:06.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:33:06.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:33:06.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:33:06.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:33:06.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:33:06.168 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:33:06.168 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:33:06.168 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:33:06.168 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:33:06.168 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:06.168 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:06.169 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:33:06.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:33:06.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:33:06.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:33:06.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:33:06.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:33:06.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:33:06.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:33:06.289 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:33:06.289 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:33:06.289 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:33:06.289 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:33:06.289 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:06.289 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:06.289 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:33:06.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:33:06.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:33:06.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: run_mgr td/ceph-helpers x 2024-08-19T11:33:06.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:33:06.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:33:06.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:33:06.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:33:06.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:33:06.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:33:07.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:33:07.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:33:07.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:33:07.338 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:33:07.338 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:07.338 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:07.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:33:07.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:33:07.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:33:07.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: run_osd td/ceph-helpers 0 2024-08-19T11:33:07.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:33:07.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:33:07.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:33:07.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:33:07.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:33:07.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:33:07.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:33:07.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bee849c4-4e6a-4081-b4d1-69c19ba794d4 2024-08-19T11:33:07.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bee849c4-4e6a-4081-b4d1-69c19ba794d4' 2024-08-19T11:33:07.380 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 bee849c4-4e6a-4081-b4d1-69c19ba794d4 2024-08-19T11:33:07.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:33:07.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzLcNm4QW/FxAAJ4NSxBD7Pndezl642I7wdw== 2024-08-19T11:33:07.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzLcNm4QW/FxAAJ4NSxBD7Pndezl642I7wdw=="}' 2024-08-19T11:33:07.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bee849c4-4e6a-4081-b4d1-69c19ba794d4 -i td/ceph-helpers/0/new.json 2024-08-19T11:33:07.717 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:33:07.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:33:07.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzLcNm4QW/FxAAJ4NSxBD7Pndezl642I7wdw== --osd-uuid bee849c4-4e6a-4081-b4d1-69c19ba794d4 2024-08-19T11:33:07.770 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:07.767+0000 7fe3100527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:33:07.795 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:07.791+0000 7fe3100527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:33:07.797 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:07.795+0000 7fe3100527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:33:07.797 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:07.795+0000 7fe3100527c0 -1 bdev(0x55abe5ea5c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:33:07.797 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:07.795+0000 7fe3100527c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:33:14.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:33:14.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:33:14.650 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:33:14.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:33:14.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:33:15.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:33:15.244 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:33:15.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:33:15.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:33:15.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:33:15.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:33:15.270 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:15.267+0000 7f15d203c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:33:15.284 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:15.283+0000 7f15d203c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:33:15.285 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:15.283+0000 7f15d203c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:33:15.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:33:15.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:33:15.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:33:15.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:33:15.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:33:15.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:15.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:33:15.669 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:33:15.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:15.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:16.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:16.566 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:16.563+0000 7f15d203c7c0 -1 Falling back to public interface 2024-08-19T11:33:17.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:17.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:17.103 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:33:17.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:33:17.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:17.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:17.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:18.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:18.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:18.529 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:33:18.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:33:18.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:18.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:18.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:19.560 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:19.559+0000 7f15d203c7c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:33:19.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:19.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:19.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:33:19.982 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:33:19.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:19.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:20.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:21.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:21.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:21.455 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:33:21.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:33:21.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:21.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:21.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:22.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:22.934 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:33:22.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:22.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:33:22.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:22.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:23.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:24.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:24.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:24.440 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:33:24.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:33:24.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:24.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:24.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:25.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:25.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:25.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:33:25.893 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:33:25.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:25.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:26.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:27.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:27.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:27.437 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:33:27.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:33:27.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:27.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:27.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:28.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:28.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:28.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:33:28.915 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:33:28.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:28.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:29.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:30.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:30.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:30.496 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:33:30.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:33:30.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:30.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:30.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:31.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:31.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:31.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:33:31.949 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:33:31.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:31.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:32.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:33.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:33.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:33.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:33:33.570 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:33:33.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:33.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:34.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:35.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:35.366 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:33:35.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:35.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:33:35.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:35.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:36.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:37.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:37.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:37.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:33:37.317 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:33:37.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:37.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:38.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:39.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:39.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:39.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:33:39.117 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:33:39.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:39.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:40.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:41.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:41.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:41.044 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:33:41.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:33:41.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:41.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:41.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:42.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:42.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:42.893 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:33:42.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:33:42.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:42.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:44.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:45.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:45.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:45.038 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:33:45.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:33:45.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:45.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:45.471 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3258931966,v1:127.0.0.1:6803/3258931966] [v2:127.0.0.1:6804/3258931966,v1:127.0.0.1:6805/3258931966] exists,up bee849c4-4e6a-4081-b4d1-69c19ba794d4 2024-08-19T11:33:45.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:33:45.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:33:45.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:33:45.473 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: get_asok_path osd.0 2024-08-19T11:33:45.473 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:33:45.473 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:33:45.473 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:33:45.473 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:45.474 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:45.474 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-osd.0.asok 2024-08-19T11:33:45.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: CEPH_ARGS= 2024-08-19T11:33:45.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.71449/ceph-osd.0.asok config get osd_max_backfills 2024-08-19T11:33:45.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-08-19T11:33:45.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2024-08-19T11:33:45.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: grep --quiet osd_max_backfills 2024-08-19T11:33:45.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:898: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2024-08-19T11:33:45.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:33:45.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:33:45.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:33:45.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:33:45.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:33:46.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:33:46.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-08-19T11:33:46.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-08-19T11:33:46.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T11:33:46.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T11:33:46.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T11:33:46.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:33:46.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:33:46.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:33:46.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:33:46.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:33:46.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:33:46.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:33:46.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T11:33:46.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T11:33:46.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:33:46.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T11:33:46.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:33:46.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:33:46.912 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:33:46.912 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:46.912 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:46.913 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:33:46.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:33:46.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:33:46.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:33:46.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:33:46.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:33:46.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:33:46.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:33:46.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:33:46.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T11:33:46.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-08-19T11:33:46.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:33:46.914 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:33:46.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T11:33:46.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-08-19T11:33:46.914 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-08-19T11:33:46.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T11:33:46.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T11:33:46.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T11:33:46.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T11:33:46.949 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:46.947+0000 7fa08f9297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:33:46.952 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:46.951+0000 7fa08f9297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:33:46.954 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:46.951+0000 7fa08f9297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:33:47.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T11:33:47.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:33:47.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:33:47.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:33:47.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:33:47.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:47.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:33:47.401 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:33:47.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:47.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:47.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:48.191 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:48.186+0000 7fa08f9297c0 -1 Falling back to public interface 2024-08-19T11:33:48.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:48.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:48.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:33:48.856 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:33:48.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:48.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:49.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:50.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:50.281 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:33:50.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:50.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:33:50.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:50.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:50.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:33:51.730 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:51.726+0000 7fa08f9297c0 -1 osd.0 6 log_to_monitors true 2024-08-19T11:33:51.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:33:51.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:33:51.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:33:51.760 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:33:51.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:33:51.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:33:52.331 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/2115376047,v1:127.0.0.1:6803/2115376047] [v2:127.0.0.1:6804/2115376047,v1:127.0.0.1:6805/2115376047] exists,up bee849c4-4e6a-4081-b4d1-69c19ba794d4 2024-08-19T11:33:52.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:33:52.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:33:52.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:33:52.332 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: get_asok_path osd.0 2024-08-19T11:33:52.332 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:33:52.332 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:33:52.332 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:33:52.332 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:52.333 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:52.333 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-osd.0.asok 2024-08-19T11:33:52.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: CEPH_ARGS= 2024-08-19T11:33:52.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.71449/ceph-osd.0.asok config get osd_max_backfills 2024-08-19T11:33:52.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-08-19T11:33:52.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:903: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-08-19T11:33:52.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd: teardown td/ceph-helpers 2024-08-19T11:33:52.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:33:52.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:33:52.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:33:52.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:33:52.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:33:52.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:33:52.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:33:52.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:33:52.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:33:52.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:33:52.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:33:52.563 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:33:52.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:33:52.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:33:52.564 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:33:52.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:33:52.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:33:52.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:33:52.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:33:52.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:33:52.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:33:52.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:33:52.572 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:33:52.573 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:52.573 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:52.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:33:52.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:33:52.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:33:52.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:33:52.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2024-08-19T11:33:52.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2024-08-19T11:33:52.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:911: test_activate_osd_after_mark_down: setup td/ceph-helpers 2024-08-19T11:33:52.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:33:52.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:33:52.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:33:52.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:33:52.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:33:52.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:33:52.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:33:52.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:33:52.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:33:52.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:33:52.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:33:52.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:33:52.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:33:52.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:33:52.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:33:52.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:33:52.579 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:33:52.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:33:52.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:33:52.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:33:52.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:33:52.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:33:52.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:33:52.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:33:52.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:33:52.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:52.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:52.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:33:52.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:33:52.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:33:52.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:33:52.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:33:52.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:52.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:52.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:33:52.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:33:52.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:33:52.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:33:52.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:33:52.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2024-08-19T11:33:52.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:33:52.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:33:52.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:33:52.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:33:52.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:33:52.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-08-19T11:33:53.067 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:33:53.067 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:33:53.067 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:33:53.068 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:33:53.068 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:53.068 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:53.068 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:33:53.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:33:53.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:33:53.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:33:53.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:33:53.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:33:53.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:33:53.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:33:53.289 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:33:53.290 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:33:53.290 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:33:53.290 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:33:53.290 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:53.290 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:53.290 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:33:53.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:33:53.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:33:53.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:33:53.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:33:53.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:33:53.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:33:53.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:33:53.398 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:33:53.398 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:33:53.398 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:33:53.399 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:33:53.399 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:53.399 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:53.399 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:33:53.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:33:53.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:33:53.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2024-08-19T11:33:53.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:33:53.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:33:53.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:33:53.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:33:53.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:33:53.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:33:54.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:33:54.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:33:54.427 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:33:54.427 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:33:54.427 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:54.427 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:54.427 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:33:54.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:33:54.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:33:54.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2024-08-19T11:33:54.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:33:54.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:33:54.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:33:54.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:33:54.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:33:54.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:33:54.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:33:54.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:33:54.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:33:54.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:33:54.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:33:54.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:33:54.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:33:54.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:33:54.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:33:54.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:33:54.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:33:54.463 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:33:54.463 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:33:54.463 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:33:54.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:33:54.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:33:54.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3e9ebcd-11e7-49ef-9b51-508c02b09c08 2024-08-19T11:33:54.465 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 f3e9ebcd-11e7-49ef-9b51-508c02b09c08 2024-08-19T11:33:54.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f3e9ebcd-11e7-49ef-9b51-508c02b09c08' 2024-08-19T11:33:54.466 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:33:54.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCiLcNmQ6jaHBAAc5syPnnvzfR96vSdaIVRKQ== 2024-08-19T11:33:54.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCiLcNmQ6jaHBAAc5syPnnvzfR96vSdaIVRKQ=="}' 2024-08-19T11:33:54.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3e9ebcd-11e7-49ef-9b51-508c02b09c08 -i td/ceph-helpers/0/new.json 2024-08-19T11:33:54.775 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:33:54.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:33:54.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCiLcNmQ6jaHBAAc5syPnnvzfR96vSdaIVRKQ== --osd-uuid f3e9ebcd-11e7-49ef-9b51-508c02b09c08 2024-08-19T11:33:54.824 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:54.818+0000 7f74c9f837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:33:54.850 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:54.846+0000 7f74c9f837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:33:54.852 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:54.850+0000 7f74c9f837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:33:54.853 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:54.850+0000 7f74c9f837c0 -1 bdev(0x55c07aa31c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:33:54.853 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:33:54.850+0000 7f74c9f837c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:34:01.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:34:01.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:34:01.187 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:34:01.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:34:01.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:34:01.831 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:34:01.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:34:01.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:34:01.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:34:01.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:34:01.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:34:01.863 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:01.858+0000 7fbe9e8237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:01.881 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:01.878+0000 7fbe9e8237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:01.883 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:01.878+0000 7fbe9e8237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:02.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:34:02.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:34:02.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:34:02.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:34:02.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:34:02.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:02.263 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:34:02.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:34:02.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:02.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:02.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:03.379 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:03.374+0000 7fbe9e8237c0 -1 Falling back to public interface 2024-08-19T11:34:03.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:03.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:03.693 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:34:03.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:34:03.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:03.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:04.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:05.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:05.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:05.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:34:05.129 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:34:05.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:05.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:05.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:06.336 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:06.334+0000 7fbe9e8237c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:34:06.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:06.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:06.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:34:06.574 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:34:06.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:06.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:07.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:08.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:08.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:08.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:34:08.211 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:34:08.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:08.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:08.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:09.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:09.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:09.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:34:09.696 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:34:09.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:09.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:10.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:11.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:11.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:11.222 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:34:11.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:34:11.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:11.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:11.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:12.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:12.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:12.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:34:12.684 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:34:12.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:12.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:13.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:14.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:14.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:14.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:34:14.246 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:34:14.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:14.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:14.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:15.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:15.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:15.719 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:34:15.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:34:15.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:15.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:16.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:17.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:17.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:17.269 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:34:17.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:34:17.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:17.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:17.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:18.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:18.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:18.736 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:34:18.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:34:18.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:18.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:19.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:20.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:20.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:20.283 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:34:20.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:34:20.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:20.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:20.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:21.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:21.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:21.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:34:21.804 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:34:21.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:21.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:22.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:23.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:23.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:23.389 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:34:23.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:34:23.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:23.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:24.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:25.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:25.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:25.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:34:25.468 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:34:25.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:25.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:26.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:27.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:27.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:27.549 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:34:27.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:34:27.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:27.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:28.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:29.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:29.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:29.225 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:34:29.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:34:29.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:29.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:30.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:31.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:31.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:31.366 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:34:31.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:34:31.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:31.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:31.862 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/228133428,v1:127.0.0.1:6803/228133428] [v2:127.0.0.1:6804/228133428,v1:127.0.0.1:6805/228133428] exists,up f3e9ebcd-11e7-49ef-9b51-508c02b09c08 2024-08-19T11:34:31.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:34:31.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:34:31.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:34:31.863 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-08-19T11:34:31.863 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:34:31.863 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:34:31.864 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:34:31.864 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:31.864 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:31.864 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-osd.0.asok 2024-08-19T11:34:31.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-08-19T11:34:31.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.71449/ceph-osd.0.asok config get osd_max_backfills 2024-08-19T11:34:31.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2024-08-19T11:34:31.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2024-08-19T11:34:31.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2024-08-19T11:34:31.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2024-08-19T11:34:31.989 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:34:31.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:34:31.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:34:31.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:34:31.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:34:33.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:34:33.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: test_activate_osd_after_mark_down: ceph osd down 0 2024-08-19T11:34:33.737 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already down. 2024-08-19T11:34:33.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: test_activate_osd_after_mark_down: wait_for_osd down 0 2024-08-19T11:34:33.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-08-19T11:34:33.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:34:33.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:34:33.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:34:33.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:33.754 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:34:33.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:34:33.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:33.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-08-19T11:34:34.222 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/228133428,v1:127.0.0.1:6803/228133428] [v2:127.0.0.1:6804/228133428,v1:127.0.0.1:6805/228133428] exists f3e9ebcd-11e7-49ef-9b51-508c02b09c08 2024-08-19T11:34:34.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:34:34.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:34:34.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:34:34.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-08-19T11:34:34.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-08-19T11:34:34.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T11:34:34.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T11:34:34.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T11:34:34.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:34:34.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:34:34.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:34:34.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:34:34.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:34:34.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:34:34.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:34:34.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T11:34:34.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T11:34:34.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:34:34.227 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T11:34:34.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:34:34.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:34:34.228 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:34:34.228 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:34.228 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:34.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:34:34.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:34:34.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:34:34.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:34:34.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:34:34.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:34:34.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:34:34.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:34:34.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:34:34.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T11:34:34.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-08-19T11:34:34.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:34:34.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T11:34:34.229 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:34:34.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-08-19T11:34:34.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-08-19T11:34:34.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T11:34:34.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T11:34:34.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T11:34:34.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T11:34:34.264 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:34.265+0000 7fbf4ee557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:34.266 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:34.265+0000 7fbf4ee557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:34.268 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:34.269+0000 7fbf4ee557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:34.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T11:34:34.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:34:34.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:34:34.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:34:34.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:34:34.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:34.637 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:34:34.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:34:34.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:34.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:35.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:35.551 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:35.549+0000 7fbf4ee557c0 -1 Falling back to public interface 2024-08-19T11:34:36.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:36.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:36.077 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:34:36.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:34:36.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:36.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:36.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:37.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:37.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:37.519 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:34:37.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:34:37.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:37.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:37.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:38.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:38.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:38.944 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:34:38.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:34:38.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:38.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:39.030 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:39.029+0000 7fbf4ee557c0 -1 osd.0 5 log_to_monitors true 2024-08-19T11:34:39.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:40.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:40.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:40.361 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:34:40.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:34:40.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:40.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:40.785 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/1663624096,v1:127.0.0.1:6803/1663624096] [v2:127.0.0.1:6804/1663624096,v1:127.0.0.1:6805/1663624096] exists,up f3e9ebcd-11e7-49ef-9b51-508c02b09c08 2024-08-19T11:34:40.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:34:40.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:34:40.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:34:40.786 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-08-19T11:34:40.786 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:34:40.786 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:34:40.787 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:34:40.787 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:40.787 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:40.787 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-osd.0.asok 2024-08-19T11:34:40.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-08-19T11:34:40.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.71449/ceph-osd.0.asok config get osd_max_backfills 2024-08-19T11:34:40.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2024-08-19T11:34:40.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-08-19T11:34:40.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2024-08-19T11:34:40.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:34:40.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:34:40.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:34:40.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:34:40.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:34:40.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:34:40.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:34:40.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:34:41.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:34:41.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:34:41.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:34:41.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:34:41.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:34:41.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:34:41.020 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:34:41.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:34:41.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:34:41.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:34:41.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:34:41.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:34:41.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:34:41.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:34:41.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:34:41.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:41.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:41.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:34:41.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:34:41.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:34:41.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:34:41.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2024-08-19T11:34:41.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:934: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2024-08-19T11:34:41.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2024-08-19T11:34:41.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:34:41.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:34:41.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:34:41.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:34:41.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:34:41.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:34:41.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:34:41.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:34:41.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:34:41.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:34:41.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:34:41.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:34:41.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:34:41.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:34:41.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:34:41.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:34:41.035 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:34:41.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:34:41.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:34:41.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:34:41.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:34:41.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:34:41.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:34:41.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:34:41.040 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:34:41.040 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:41.040 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:41.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:34:41.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:34:41.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:34:41.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:34:41.041 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:34:41.041 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:41.041 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:41.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:34:41.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:34:41.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:34:41.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:34:41.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:34:41.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2024-08-19T11:34:41.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:34:41.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:34:41.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:34:41.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:34:41.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:34:41.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-08-19T11:34:41.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:34:41.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:34:41.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:34:41.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:34:41.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:41.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:41.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:34:41.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:34:41.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:34:41.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:34:41.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:34:41.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:34:41.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:34:41.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:34:41.778 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:34:41.778 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:34:41.778 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:34:41.779 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:34:41.779 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:41.779 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:41.779 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:34:41.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:34:41.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:34:41.892 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:34:41.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:34:41.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:34:41.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:34:41.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:34:41.894 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:34:41.894 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:34:41.894 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:34:41.894 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:34:41.894 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:41.895 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:41.895 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:34:41.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:34:41.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:34:42.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:939: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2024-08-19T11:34:42.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:34:42.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:34:42.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:34:42.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:34:42.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:34:42.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:34:42.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:34:42.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:34:42.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:34:42.905 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:34:42.905 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:42.905 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:42.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:34:42.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:34:42.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:34:42.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-08-19T11:34:42.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:34:42.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:34:42.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:34:42.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:34:42.936 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:34:42.937 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:42.937 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:42.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:34:42.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:34:42.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:34:42.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:34:42.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:34:42.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:34:42.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:34:42.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:34:42.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:34:42.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:34:42.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:34:42.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:34:42.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-08-19T11:34:42.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:34:42.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:34:42.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b5f1af69-ea78-4b8f-917e-05aeacd0a01f 2024-08-19T11:34:42.939 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 b5f1af69-ea78-4b8f-917e-05aeacd0a01f 2024-08-19T11:34:42.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b5f1af69-ea78-4b8f-917e-05aeacd0a01f' 2024-08-19T11:34:42.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:34:42.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSLcNm/b4OORAAQD/PYKoo761kUP/4OvQFng== 2024-08-19T11:34:42.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSLcNm/b4OORAAQD/PYKoo761kUP/4OvQFng=="}' 2024-08-19T11:34:42.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b5f1af69-ea78-4b8f-917e-05aeacd0a01f -i td/ceph-helpers/0/new.json 2024-08-19T11:34:43.271 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:34:43.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:34:43.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQDSLcNm/b4OORAAQD/PYKoo761kUP/4OvQFng== --osd-uuid b5f1af69-ea78-4b8f-917e-05aeacd0a01f 2024-08-19T11:34:43.317 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:43.317+0000 7fddeffc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:43.347 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:43.345+0000 7fddeffc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:43.349 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:43.349+0000 7fddeffc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:43.349 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:43.349+0000 7fddeffc07c0 -1 bdev(0x55ce8c850800 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:34:43.350 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:43.349+0000 7fddeffc07c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:34:49.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:34:49.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:34:49.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:34:49.907 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:34:49.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:34:50.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:34:50.487 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:34:50.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-08-19T11:34:50.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:34:50.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:34:50.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:34:50.520 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:50.521+0000 7f151a7e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:50.544 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:50.545+0000 7f151a7e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:50.545+0000 7f151a7e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:50.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:34:50.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:34:50.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:34:50.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:34:50.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:34:50.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:50.931 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:34:50.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:34:50.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:50.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:51.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:51.639 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:51.637+0000 7f151a7e37c0 -1 Falling back to public interface 2024-08-19T11:34:52.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:52.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:52.377 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:34:52.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:34:52.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:52.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:52.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:53.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:53.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:53.793 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:34:53.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:34:53.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:53.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:54.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:54.584 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:54.585+0000 7f151a7e37c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:34:55.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:55.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:55.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:34:55.278 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:34:55.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:55.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:55.789 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2626547238,v1:127.0.0.1:6803/2626547238] [v2:127.0.0.1:6804/2626547238,v1:127.0.0.1:6805/2626547238] exists,up b5f1af69-ea78-4b8f-917e-05aeacd0a01f 2024-08-19T11:34:55.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:34:55.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:34:55.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:34:55.790 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-08-19T11:34:55.791 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:34:55.791 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:34:55.791 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:34:55.791 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:55.791 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:55.792 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-osd.0.asok 2024-08-19T11:34:55.792 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-08-19T11:34:55.792 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.71449/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-08-19T11:34:55.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-08-19T11:34:55.909 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-08-19T11:34:55.909 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:34:55.909 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:34:55.910 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:34:55.910 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:55.910 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:55.910 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-osd.0.asok 2024-08-19T11:34:55.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-08-19T11:34:55.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.71449/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-08-19T11:34:56.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-08-19T11:34:56.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2024-08-19T11:34:56.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:34:56.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:34:56.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:34:56.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:34:56.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:34:56.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:34:56.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:950: test_activate_osd_skip_benchmark: ceph osd down 0 2024-08-19T11:34:56.770 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already down. 2024-08-19T11:34:56.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: wait_for_osd down 0 2024-08-19T11:34:56.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-08-19T11:34:56.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:34:56.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:34:56.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:34:56.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:56.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:34:56.789 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:34:56.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:56.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-08-19T11:34:57.264 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/2626547238,v1:127.0.0.1:6803/2626547238] [v2:127.0.0.1:6804/2626547238,v1:127.0.0.1:6805/2626547238] exists b5f1af69-ea78-4b8f-917e-05aeacd0a01f 2024-08-19T11:34:57.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:34:57.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:34:57.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:34:57.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:955: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-08-19T11:34:57.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-08-19T11:34:57.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T11:34:57.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T11:34:57.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T11:34:57.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:34:57.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:34:57.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:34:57.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:34:57.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:34:57.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:34:57.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:34:57.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T11:34:57.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T11:34:57.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:34:57.268 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:34:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T11:34:57.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-08-19T11:34:57.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:34:57.270 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:34:57.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T11:34:57.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-08-19T11:34:57.271 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-08-19T11:34:57.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T11:34:57.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T11:34:57.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T11:34:57.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T11:34:57.294 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:57.293+0000 7f3794aa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:57.297 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:57.297+0000 7f3794aa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:57.298 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:57.297+0000 7f3794aa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:34:57.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T11:34:57.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:34:57.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:34:57.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:34:57.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:34:57.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:57.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:34:57.706 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:34:57.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:57.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:58.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:34:58.815 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:34:58.813+0000 7f3794aa37c0 -1 Falling back to public interface 2024-08-19T11:34:59.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:34:59.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:34:59.160 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:34:59.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:34:59.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:34:59.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:34:59.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:00.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:00.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:00.579 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:35:00.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:35:00.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:00.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:01.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:02.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:02.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:02.038 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:35:02.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:35:02.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:02.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:02.331 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:02.329+0000 7f3794aa37c0 -1 osd.0 6 log_to_monitors true 2024-08-19T11:35:02.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:03.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:03.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:03.544 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:35:03.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:35:03.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:03.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:04.027 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/1957656617,v1:127.0.0.1:6803/1957656617] [v2:127.0.0.1:6804/1957656617,v1:127.0.0.1:6805/1957656617] exists,up b5f1af69-ea78-4b8f-917e-05aeacd0a01f 2024-08-19T11:35:04.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:35:04.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:35:04.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:35:04.029 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-08-19T11:35:04.029 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:35:04.029 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:35:04.029 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:35:04.029 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:04.029 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:04.029 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-osd.0.asok 2024-08-19T11:35:04.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-08-19T11:35:04.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.71449/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-08-19T11:35:04.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-08-19T11:35:04.146 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-08-19T11:35:04.147 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:35:04.147 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:35:04.147 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:35:04.147 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:04.147 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:04.147 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-osd.0.asok 2024-08-19T11:35:04.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-08-19T11:35:04.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.71449/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-08-19T11:35:04.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-08-19T11:35:04.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-08-19T11:35:04.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-08-19T11:35:04.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2024-08-19T11:35:04.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:35:04.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:35:04.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:35:04.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:35:04.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:35:04.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:35:04.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:35:04.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:35:04.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:35:04.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:35:04.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:35:04.373 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:35:04.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:35:04.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:35:04.374 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:35:04.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:04.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:35:04.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:35:04.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:04.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:35:04.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:35:04.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:35:04.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:35:04.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:04.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:04.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:35:04.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:35:04.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:35:04.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:35:04.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_destroy_osd td/ceph-helpers 2024-08-19T11:35:04.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:796: test_destroy_osd: local dir=td/ceph-helpers 2024-08-19T11:35:04.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:798: test_destroy_osd: setup td/ceph-helpers 2024-08-19T11:35:04.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:35:04.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:35:04.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:35:04.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:35:04.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:35:04.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:35:04.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:35:04.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:35:04.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:35:04.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:35:04.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:35:04.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:35:04.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:35:04.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:35:04.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:35:04.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:35:04.385 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:35:04.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:04.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:35:04.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:35:04.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:04.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:35:04.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:35:04.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:35:04.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:35:04.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:04.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:04.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:35:04.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:35:04.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:35:04.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:35:04.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:35:04.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:04.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:04.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:35:04.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:35:04.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:35:04.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:35:04.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:35:04.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:799: test_destroy_osd: run_mon td/ceph-helpers a 2024-08-19T11:35:04.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:35:04.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:35:04.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:35:04.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:35:04.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:35:04.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-08-19T11:35:04.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:35:04.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:35:04.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:35:04.877 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:35:04.877 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:04.877 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:04.878 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:35:04.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:35:05.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:35:05.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:35:05.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:35:05.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:35:05.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:35:05.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:35:05.099 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:35:05.099 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:35:05.099 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:35:05.099 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:35:05.099 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:05.099 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:05.099 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:35:05.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:35:05.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:35:05.216 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:35:05.216 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:35:05.216 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:35:05.216 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:35:05.217 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:35:05.217 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:35:05.217 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:35:05.217 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:35:05.218 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:35:05.218 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:05.218 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:05.218 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:35:05.218 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:35:05.218 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:35:05.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:800: test_destroy_osd: run_mgr td/ceph-helpers x 2024-08-19T11:35:05.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:35:05.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:35:05.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:35:05.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:35:05.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:35:05.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:35:06.243 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:35:06.243 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:35:06.243 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:35:06.243 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:35:06.243 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:06.243 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:06.243 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:35:06.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:35:06.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:35:06.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:801: test_destroy_osd: run_osd td/ceph-helpers 0 2024-08-19T11:35:06.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:35:06.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:35:06.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:35:06.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:35:06.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:35:06.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:35:06.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:35:06.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:35:06.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:35:06.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:35:06.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:35:06.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:35:06.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:35:06.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:35:06.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:35:06.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:35:06.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:35:06.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:35:06.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:35:06.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:35:06.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:35:06.285 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:35:06.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f151bb71-d953-4cd6-88f1-f9954c3e6a60 2024-08-19T11:35:06.286 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 f151bb71-d953-4cd6-88f1-f9954c3e6a60 2024-08-19T11:35:06.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f151bb71-d953-4cd6-88f1-f9954c3e6a60' 2024-08-19T11:35:06.286 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:35:06.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDqLcNmpmUnEhAAIgeFqA22GhOg9we1Ts6jmA== 2024-08-19T11:35:06.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDqLcNmpmUnEhAAIgeFqA22GhOg9we1Ts6jmA=="}' 2024-08-19T11:35:06.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f151bb71-d953-4cd6-88f1-f9954c3e6a60 -i td/ceph-helpers/0/new.json 2024-08-19T11:35:06.633 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:35:06.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:35:06.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDqLcNmpmUnEhAAIgeFqA22GhOg9we1Ts6jmA== --osd-uuid f151bb71-d953-4cd6-88f1-f9954c3e6a60 2024-08-19T11:35:06.679 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:06.677+0000 7f7c7fa597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:35:06.682 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:06.681+0000 7f7c7fa597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:35:06.684 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:06.681+0000 7f7c7fa597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:35:06.684 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:06.685+0000 7f7c7fa597c0 -1 bdev(0x5630a8a69c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:35:06.684 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:06.685+0000 7f7c7fa597c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:35:13.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:35:13.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:35:13.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:35:13.400 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:35:13.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:35:13.968 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:35:13.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:35:13.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:35:13.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:35:13.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:35:13.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:35:14.000 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:14.001+0000 7f5c6384e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:35:14.024 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:14.025+0000 7f5c6384e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:35:14.025 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:14.025+0000 7f5c6384e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:35:14.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:35:14.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:35:14.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:35:14.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:35:14.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:35:14.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:14.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:35:14.400 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:35:14.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:14.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:14.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:15.291 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:15.289+0000 7f5c6384e7c0 -1 Falling back to public interface 2024-08-19T11:35:15.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:15.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:15.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:35:15.843 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:35:15.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:15.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:16.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:17.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:17.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:17.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:35:17.268 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:35:17.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:17.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:17.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:18.287 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:18.285+0000 7f5c6384e7c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:35:18.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:18.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:18.687 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:35:18.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:35:18.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:18.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:19.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:20.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:20.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:20.132 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:35:20.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:35:20.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:20.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:20.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:21.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:21.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:21.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:35:21.598 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:35:21.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:21.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:22.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:23.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:23.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:23.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:35:23.173 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:35:23.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:23.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:23.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:24.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:24.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:24.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:35:24.642 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:35:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:25.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:26.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:26.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:26.221 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:35:26.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:35:26.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:26.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:26.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:27.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:27.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:27.704 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:35:27.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:35:27.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:27.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:28.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:29.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:29.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:29.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:35:29.249 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:35:29.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:29.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:29.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:30.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:30.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:30.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:35:30.709 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:35:30.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:30.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:31.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:32.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:32.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:32.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:35:32.539 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:35:32.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:32.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:33.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:34.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:34.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:34.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:35:34.084 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:35:34.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:34.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:34.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:35.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:35.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:35.700 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:35:35.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:35:35.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:35.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:36.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:37.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:37.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:37.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:35:37.636 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:35:37.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:37.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:38.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:39.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:39.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:39.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:35:39.647 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:35:39.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:39.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:40.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:41.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:41.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:41.458 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:35:41.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:35:41.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:41.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:42.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:43.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:43.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:43.686 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:35:43.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:35:43.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:43.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:44.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:35:45.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:35:45.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:35:45.147 INFO:tasks.workunit.client.0.smithi130.stdout:19 2024-08-19T11:35:45.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T11:35:45.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:35:45.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:35:45.592 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1078074691,v1:127.0.0.1:6803/1078074691] [v2:127.0.0.1:6804/1078074691,v1:127.0.0.1:6805/1078074691] exists,up f151bb71-d953-4cd6-88f1-f9954c3e6a60 2024-08-19T11:35:45.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:35:45.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:35:45.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:35:45.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:802: test_destroy_osd: destroy_osd td/ceph-helpers 0 2024-08-19T11:35:45.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/ceph-helpers 2024-08-19T11:35:45.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=0 2024-08-19T11:35:45.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.0 2024-08-19T11:35:46.214 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already out. 2024-08-19T11:35:46.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-08-19T11:35:46.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:35:46.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:35:46.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:35:46.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:35:46.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:35:46.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:35:46.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.0 2024-08-19T11:35:46.969 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already down. 2024-08-19T11:35:46.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2024-08-19T11:35:47.517 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 does not exist 2024-08-19T11:35:47.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/ceph-helpers/0 2024-08-19T11:35:47.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers/0 2024-08-19T11:35:47.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:35:47.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers/0 KILL 2024-08-19T11:35:47.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:35:47.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:35:47.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:35:47.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:35:47.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:35:47.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:35:47.538 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:35:47.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:35:47.539 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:35:47.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:35:47.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:35:47.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:35:47.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:47.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:35:47.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:35:47.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:47.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:35:47.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:35:47.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers/0 2024-08-19T11:35:47.548 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:35:47.548 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:47.548 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:47.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:35:47.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:35:47.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:35:47.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/ceph-helpers/0 2024-08-19T11:35:47.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: ceph osd dump 2024-08-19T11:35:47.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: grep 'osd. ' 2024-08-19T11:35:47.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:804: test_destroy_osd: teardown td/ceph-helpers 2024-08-19T11:35:47.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:35:47.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:35:47.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:35:47.986 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:35:47.986 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:35:47.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:35:47.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:35:47.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:35:48.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:35:48.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:35:48.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:35:48.096 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:35:48.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:35:48.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:35:48.097 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:35:48.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:48.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:35:48.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:35:48.099 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:48.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:35:48.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:35:48.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:35:48.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:35:48.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:48.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:48.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:35:48.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:35:48.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:35:48.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:35:48.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_display_logs td/ceph-helpers 2024-08-19T11:35:48.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2119: test_display_logs: local dir=td/ceph-helpers 2024-08-19T11:35:48.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: test_display_logs: setup td/ceph-helpers 2024-08-19T11:35:48.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:35:48.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:35:48.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:35:48.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:35:48.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:35:48.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:35:48.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:35:48.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:35:48.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:35:48.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:35:48.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:35:48.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:35:48.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:35:48.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:35:48.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:35:48.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:35:48.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:35:48.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:48.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:35:48.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:35:48.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:48.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:35:48.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:35:48.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:35:48.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:35:48.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:48.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:48.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:35:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:35:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:35:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:35:48.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:35:48.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:48.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:48.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:35:48.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:35:48.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:35:48.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:35:48.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:35:48.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: test_display_logs: run_mon td/ceph-helpers a 2024-08-19T11:35:48.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:35:48.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:35:48.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:35:48.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:35:48.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:35:48.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-08-19T11:35:48.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:35:48.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:35:48.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:35:48.593 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:35:48.593 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:48.593 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:48.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:35:48.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:35:48.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:35:48.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:35:48.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:35:48.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:35:48.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:35:48.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:35:48.814 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:35:48.814 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:35:48.814 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:35:48.814 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:35:48.814 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:48.814 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:48.814 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:35:48.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:35:48.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:35:48.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:35:48.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:35:48.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:35:48.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:35:48.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:35:48.929 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:35:48.929 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:35:48.929 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:35:48.929 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:35:48.929 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:48.930 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:48.930 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:35:48.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:35:48.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:35:49.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: test_display_logs: kill_daemons td/ceph-helpers 2024-08-19T11:35:49.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:35:49.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:35:49.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:35:49.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:35:49.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:35:50.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:35:50.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: test_display_logs: display_logs td/ceph-helpers 2024-08-19T11:35:50.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: display_logs: local dir=td/ceph-helpers 2024-08-19T11:35:50.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2024-08-19T11:35:50.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: display_logs: read file 2024-08-19T11:35:50.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2024-08-19T11:35:50.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: display_logs: cat td/ceph-helpers/mon.a.log 2024-08-19T11:35:50.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: display_logs: read file 2024-08-19T11:35:50.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2024-08-19T11:35:50.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: test_display_logs: teardown td/ceph-helpers 2024-08-19T11:35:50.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:35:50.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:35:50.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:35:50.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:35:50.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:35:50.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:35:50.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:35:50.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:35:50.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:35:50.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:35:50.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:35:50.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:35:50.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:35:50.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:35:50.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:35:50.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:50.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:35:50.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:35:50.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:50.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:35:50.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:35:50.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:35:50.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:35:50.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:50.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:50.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:35:50.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:35:50.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:35:50.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:35:50.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2024-08-19T11:35:50.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2024-08-19T11:35:50.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2092: test_erasure_code_plugin_exists: setup td/ceph-helpers 2024-08-19T11:35:50.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:35:50.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:35:50.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:35:50.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:35:50.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:35:50.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:35:50.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:35:50.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:35:50.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:35:50.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:35:50.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:35:50.367 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:35:50.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:35:50.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:35:50.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:35:50.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:35:50.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:35:50.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:50.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:35:50.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:35:50.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:50.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:35:50.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:35:50.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:35:50.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:35:50.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:50.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:50.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:35:50.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:35:50.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:35:50.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:35:50.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:35:50.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:50.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:50.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:35:50.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:35:50.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:35:50.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:35:50.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:35:50.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2093: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2024-08-19T11:35:50.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:35:50.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:35:50.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:35:50.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:35:50.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:35:50.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-08-19T11:35:50.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:35:50.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:35:50.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:35:50.874 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:35:50.874 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:50.874 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:50.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:35:50.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:35:51.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:35:51.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:35:51.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:35:51.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:35:51.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:35:51.087 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:35:51.088 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:35:51.088 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:35:51.088 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:35:51.088 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:35:51.088 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:51.088 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:51.088 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:35:51.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:35:51.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:35:51.206 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:35:51.206 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:35:51.206 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:35:51.206 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:35:51.207 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:35:51.207 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:35:51.207 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:35:51.207 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:35:51.208 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:35:51.208 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:51.208 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:51.208 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:35:51.208 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:35:51.208 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:35:51.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2094: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2024-08-19T11:35:51.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:35:51.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:35:51.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:35:51.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:35:51.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:35:51.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:35:52.216 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:35:52.216 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:35:52.216 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:35:52.216 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:35:52.216 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:52.217 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:52.217 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:35:52.218 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:35:52.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:35:52.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2095: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2024-08-19T11:35:52.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: local plugin=jerasure 2024-08-19T11:35:52.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: local status 2024-08-19T11:35:52.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: erasure_code_plugin_exists: local grepstr 2024-08-19T11:35:52.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: local s 2024-08-19T11:35:52.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: erasure_code_plugin_exists: case `uname` in 2024-08-19T11:35:52.254 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: erasure_code_plugin_exists: uname 2024-08-19T11:35:52.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2024-08-19T11:35:52.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2024-08-19T11:35:52.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: s= 2024-08-19T11:35:52.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: erasure_code_plugin_exists: local status=0 2024-08-19T11:35:52.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-08-19T11:35:52.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-08-19T11:35:53.101 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile TESTPROFILE does not exist 2024-08-19T11:35:53.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: erasure_code_plugin_exists: return 0 2024-08-19T11:35:53.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2024-08-19T11:35:53.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: local plugin=FAKE 2024-08-19T11:35:53.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: local status 2024-08-19T11:35:53.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: erasure_code_plugin_exists: local grepstr 2024-08-19T11:35:53.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: local s 2024-08-19T11:35:53.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: erasure_code_plugin_exists: case `uname` in 2024-08-19T11:35:53.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: erasure_code_plugin_exists: uname 2024-08-19T11:35:53.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2024-08-19T11:35:53.119 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2024-08-19T11:35:53.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so): /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2024-08-19T11:35:53.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: erasure_code_plugin_exists: local status=5 2024-08-19T11:35:53.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2024-08-19T11:35:53.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so):' /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2024-08-19T11:35:53.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2024-08-19T11:35:53.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: erasure_code_plugin_exists: return 5 2024-08-19T11:35:53.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2024-08-19T11:35:53.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:35:53.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:35:53.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:35:53.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:35:53.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:35:53.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:35:53.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:35:53.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:35:53.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:35:53.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:35:53.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:35:53.466 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:35:53.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:35:53.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:35:53.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:35:53.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:53.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:35:53.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:35:53.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:53.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:35:53.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:35:53.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:35:53.473 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:35:53.473 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:53.473 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:53.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:35:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:35:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:35:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:35:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_expect_failure td/ceph-helpers 2024-08-19T11:35:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2000: test_expect_failure: local dir=td/ceph-helpers 2024-08-19T11:35:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: test_expect_failure: setup td/ceph-helpers 2024-08-19T11:35:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:35:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:35:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:35:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:35:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:35:53.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:35:53.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:35:53.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:35:53.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:35:53.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:35:53.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:35:53.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:35:53.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:35:53.478 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:35:53.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:35:53.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:35:53.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:35:53.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:53.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:35:53.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:35:53.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:53.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:35:53.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:35:53.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:35:53.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:35:53.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:53.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:53.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:35:53.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:35:53.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:35:53.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:35:53.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:35:53.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:53.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:53.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:35:53.489 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:35:53.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:35:53.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:35:53.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:35:53.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2003: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2024-08-19T11:35:53.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: local dir=td/ceph-helpers 2024-08-19T11:35:53.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: shift 2024-08-19T11:35:53.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: local expected=FAIL 2024-08-19T11:35:53.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: shift 2024-08-19T11:35:53.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: local success 2024-08-19T11:35:53.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: expect_failure: bash -c 'echo FAIL ; exit 1' 2024-08-19T11:35:53.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: expect_failure: success=false 2024-08-19T11:35:53.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: false 2024-08-19T11:35:53.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-08-19T11:35:53.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1995: expect_failure: return 0 2024-08-19T11:35:53.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2024-08-19T11:35:53.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: local dir=td/ceph-helpers 2024-08-19T11:35:53.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: shift 2024-08-19T11:35:53.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: local expected=FAIL 2024-08-19T11:35:53.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: shift 2024-08-19T11:35:53.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: local success 2024-08-19T11:35:53.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: expect_failure: bash -c 'echo FAIL ; exit 0' 2024-08-19T11:35:53.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: success=true 2024-08-19T11:35:53.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: true 2024-08-19T11:35:53.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: expect_failure: cat td/ceph-helpers/out 2024-08-19T11:35:53.494 INFO:tasks.workunit.client.0.smithi130.stderr:FAIL 2024-08-19T11:35:53.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: expect_failure: return 1 2024-08-19T11:35:53.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-08-19T11:35:53.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2008: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2024-08-19T11:35:53.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: local dir=td/ceph-helpers 2024-08-19T11:35:53.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: shift 2024-08-19T11:35:53.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: local expected=FAIL 2024-08-19T11:35:53.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: shift 2024-08-19T11:35:53.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: local success 2024-08-19T11:35:53.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2024-08-19T11:35:53.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: expect_failure: success=false 2024-08-19T11:35:53.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: false 2024-08-19T11:35:53.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-08-19T11:35:53.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: expect_failure: cat td/ceph-helpers/out 2024-08-19T11:35:53.499 INFO:tasks.workunit.client.0.smithi130.stderr:UNEXPECTED 2024-08-19T11:35:53.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: expect_failure: return 1 2024-08-19T11:35:53.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-08-19T11:35:53.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: test_expect_failure: teardown td/ceph-helpers 2024-08-19T11:35:53.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:35:53.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:35:53.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:35:53.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:35:53.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:35:53.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:35:53.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:35:53.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:35:53.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:35:53.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:35:53.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:35:53.505 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:35:53.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:35:53.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:35:53.506 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:35:53.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:53.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:35:53.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:35:53.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:53.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:35:53.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:35:53.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:35:53.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:35:53.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:53.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:53.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:35:53.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:35:53.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:35:53.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:35:53.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_flush_pg_stats td/ceph-helpers 2024-08-19T11:35:53.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: test_flush_pg_stats: local dir=td/ceph-helpers 2024-08-19T11:35:53.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: test_flush_pg_stats: setup td/ceph-helpers 2024-08-19T11:35:53.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:35:53.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:35:53.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:35:53.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:35:53.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:35:53.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:35:53.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:35:53.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:35:53.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:35:53.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:35:53.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:35:53.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:35:53.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:35:53.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:35:53.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:35:53.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:35:53.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:35:53.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:53.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:35:53.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:35:53.519 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:35:53.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:35:53.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:35:53.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:35:53.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:35:53.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:53.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:53.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:35:53.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:35:53.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:35:53.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:35:53.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:35:53.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:53.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:53.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:35:53.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:35:53.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:35:53.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:35:53.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:35:53.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:35:53.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:35:53.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:35:53.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:35:53.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:35:53.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:35:53.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:35:54.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:35:54.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:35:54.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:35:54.028 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:35:54.028 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:54.028 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:54.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:35:54.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:35:54.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:35:54.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:35:54.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:35:54.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:35:54.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:35:54.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:35:54.249 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:35:54.249 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:35:54.249 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:35:54.249 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:35:54.249 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:54.250 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:54.250 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:35:54.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:35:54.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:35:54.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:35:54.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:35:54.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:35:54.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:35:54.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:35:54.365 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:35:54.365 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:35:54.365 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:35:54.366 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:35:54.366 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:54.366 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:54.366 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:35:54.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:35:54.367 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:35:54.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: test_flush_pg_stats: run_mgr td/ceph-helpers x 2024-08-19T11:35:54.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:35:54.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:35:54.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:35:54.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:35:54.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:35:54.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:35:55.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:35:55.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:35:55.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:35:55.402 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:35:55.403 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:55.403 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:55.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:35:55.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:35:55.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:35:55.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: test_flush_pg_stats: run_osd td/ceph-helpers 0 2024-08-19T11:35:55.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:35:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:35:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:35:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:35:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:35:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:35:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:35:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:35:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:35:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:35:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:35:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:35:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:35:55.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:35:55.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:35:55.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:35:55.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:35:55.443 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:35:55.443 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:35:55.443 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:35:55.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:35:55.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:35:55.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:35:55.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:35:55.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:35:55.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:35:55.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:35:55.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:35:55.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:35:55.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:35:55.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:35:55.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:35:55.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:35:55.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:35:55.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:35:55.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=870f6b04-0d6c-4433-867c-3942392d3698 2024-08-19T11:35:55.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 870f6b04-0d6c-4433-867c-3942392d3698' 2024-08-19T11:35:55.445 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 870f6b04-0d6c-4433-867c-3942392d3698 2024-08-19T11:35:55.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:35:55.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbLsNmq/qZGxAATuBAcT4YCH3Ry0Wp6rZyXw== 2024-08-19T11:35:55.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbLsNmq/qZGxAATuBAcT4YCH3Ry0Wp6rZyXw=="}' 2024-08-19T11:35:55.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 870f6b04-0d6c-4433-867c-3942392d3698 -i td/ceph-helpers/0/new.json 2024-08-19T11:35:55.783 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:35:55.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:35:55.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbLsNmq/qZGxAATuBAcT4YCH3Ry0Wp6rZyXw== --osd-uuid 870f6b04-0d6c-4433-867c-3942392d3698 2024-08-19T11:35:55.832 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:55.832+0000 7f22f466c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:35:55.861 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:55.860+0000 7f22f466c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:35:55.862 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:55.860+0000 7f22f466c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:35:55.863 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:55.860+0000 7f22f466c7c0 -1 bdev(0x55c39b79a800 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:35:55.863 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:35:55.860+0000 7f22f466c7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:36:02.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:36:02.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:36:02.357 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:36:02.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:36:02.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:36:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:36:02.928 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:36:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:36:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:36:02.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:36:02.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:36:02.955 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:02.952+0000 7f24b1c087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:36:02.965 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:02.964+0000 7f24b1c087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:36:02.966 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:02.964+0000 7f24b1c087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:36:03.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:36:03.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:36:03.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:36:03.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:36:03.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:36:03.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:03.376 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:36:03.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:36:03.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:03.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:03.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:04.070 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:04.068+0000 7f24b1c087c0 -1 Falling back to public interface 2024-08-19T11:36:04.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:04.782 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:36:04.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:04.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:36:04.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:04.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:05.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:06.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:06.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:06.205 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:36:06.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:36:06.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:06.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:06.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:07.014 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:07.012+0000 7f24b1c087c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:36:07.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:07.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:07.648 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:36:07.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:36:07.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:07.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:08.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:09.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:09.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:09.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:36:09.137 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:36:09.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:09.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:09.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:10.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:10.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:10.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:36:10.579 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:36:10.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:10.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:11.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:12.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:12.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:12.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:36:12.172 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:36:12.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:12.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:12.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:13.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:13.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:13.614 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:36:13.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:36:13.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:13.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:14.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:15.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:15.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:15.165 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:36:15.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:36:15.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:15.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:15.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:16.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:16.621 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:36:16.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:16.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:36:16.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:16.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:17.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:18.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:18.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:18.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:36:18.193 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:36:18.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:18.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:18.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:19.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:19.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:19.633 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:36:19.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:36:19.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:19.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:20.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:21.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:21.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:21.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:36:21.179 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:36:21.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:21.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:22.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:23.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:23.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:23.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:36:23.323 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:36:23.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:23.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:24.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:25.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:25.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:25.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:36:25.248 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:36:25.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:25.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:26.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:27.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:27.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:27.071 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:36:27.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:36:27.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:27.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:27.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:28.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:28.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:28.912 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:36:28.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:36:28.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:28.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:29.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:30.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:30.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:30.719 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:36:30.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:36:30.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:30.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:31.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:32.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:32.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:32.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:36:32.375 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:36:32.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:32.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:32.862 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3151900181,v1:127.0.0.1:6803/3151900181] [v2:127.0.0.1:6804/3151900181,v1:127.0.0.1:6805/3151900181] exists,up 870f6b04-0d6c-4433-867c-3942392d3698 2024-08-19T11:36:32.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:36:32.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:36:32.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:36:32.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: test_flush_pg_stats: create_rbd_pool 2024-08-19T11:36:32.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:36:33.386 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:36:33.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:36:33.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:36:33.958 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:36:33.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:36:34.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:36:36.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: test_flush_pg_stats: rados -p rbd put obj /etc/group 2024-08-19T11:36:36.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: test_flush_pg_stats: flush_pg_stats 2024-08-19T11:36:36.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:36:36.572 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:36:37.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:36:37.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:36:37.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:36:37.084 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:36:37.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836487 2024-08-19T11:36:37.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836487 2024-08-19T11:36:37.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836487' 2024-08-19T11:36:37.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:36:37.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-08-19T11:36:37.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:36:37.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:36:37.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836487 2024-08-19T11:36:37.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:36:37.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836487 2024-08-19T11:36:37.246 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836487 2024-08-19T11:36:37.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-08-19T11:36:37.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:36:37.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836485 -lt 21474836487 2024-08-19T11:36:37.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:36:38.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:36:38.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:36:39.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836487 -lt 21474836487 2024-08-19T11:36:39.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2024-08-19T11:36:39.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: test_flush_pg_stats: ceph df detail --format=json 2024-08-19T11:36:39.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2024-08-19T11:36:39.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: test_flush_pg_stats: stored=6867 2024-08-19T11:36:39.711 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: test_flush_pg_stats: ceph df detail --format=json 2024-08-19T11:36:39.711 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2024-08-19T11:36:40.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: test_flush_pg_stats: stored_raw=6867 2024-08-19T11:36:40.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: test_flush_pg_stats: test 6867 -gt 0 2024-08-19T11:36:40.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: test_flush_pg_stats: test 6867 == 6867 2024-08-19T11:36:40.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: test_flush_pg_stats: teardown td/ceph-helpers 2024-08-19T11:36:40.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:36:40.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:36:40.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:36:40.305 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:36:40.305 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:36:40.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:36:40.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:36:40.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:36:40.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:36:40.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:36:40.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:36:40.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:36:40.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:36:40.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:36:40.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:36:40.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:36:40.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:36:40.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:36:40.415 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:36:40.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:36:40.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:36:40.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:36:40.422 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:36:40.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:36:40.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:36:40.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:36:40.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:36:40.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:36:40.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:36:40.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_get_config td/ceph-helpers 2024-08-19T11:36:40.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: local dir=td/ceph-helpers 2024-08-19T11:36:40.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: test_get_config: setup td/ceph-helpers 2024-08-19T11:36:40.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:36:40.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:36:40.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:36:40.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:36:40.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:36:40.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:36:40.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:36:40.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:36:40.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:36:40.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:36:40.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:36:40.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:36:40.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:36:40.430 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:36:40.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:36:40.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:36:40.430 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:36:40.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:36:40.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:36:40.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:36:40.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:36:40.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:36:40.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:36:40.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:36:40.434 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:36:40.434 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:36:40.434 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:36:40.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:36:40.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:36:40.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:36:40.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:36:40.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:36:40.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:36:40.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:36:40.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:36:40.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:36:40.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:36:40.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:36:40.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:36:40.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1140: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:36:40.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:36:40.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:36:40.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:36:40.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:36:40.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:36:40.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:36:40.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:36:40.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:36:40.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:36:40.918 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:36:40.918 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:36:40.919 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:36:40.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:36:40.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:36:41.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:36:41.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:36:41.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:36:41.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:36:41.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:36:41.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:36:41.123 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:36:41.123 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:36:41.123 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:36:41.124 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:36:41.124 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:36:41.124 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:36:41.124 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:36:41.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:36:41.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:36:41.233 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:36:41.233 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:36:41.234 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:36:41.234 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:36:41.235 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:36:41.235 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:36:41.235 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:36:41.235 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:36:41.235 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:36:41.235 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:36:41.235 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:36:41.235 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:36:41.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:36:41.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:36:41.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: get_config mon a osd_pool_default_size 2024-08-19T11:36:41.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:36:41.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:36:41.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_pool_default_size 2024-08-19T11:36:41.341 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_pool_default_size 2024-08-19T11:36:41.341 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:36:41.341 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:36:41.342 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:36:41.342 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:36:41.342 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:36:41.342 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:36:41.342 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:36:41.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:36:41.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get osd_pool_default_size 2024-08-19T11:36:41.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: test 1 = 1 2024-08-19T11:36:41.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: test_get_config: run_mgr td/ceph-helpers x 2024-08-19T11:36:41.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:36:41.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:36:41.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:36:41.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:36:41.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:36:41.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:36:42.382 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:36:42.382 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:36:42.382 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:36:42.382 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:36:42.383 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:36:42.383 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:36:42.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:36:42.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:36:42.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:36:42.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1143: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2024-08-19T11:36:42.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:36:42.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:36:42.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:36:42.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:36:42.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:36:42.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:36:42.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:36:42.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:36:42.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:36:42.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:36:42.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:36:42.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:36:42.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:36:42.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:36:42.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:36:42.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:36:42.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:36:42.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:36:42.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:36:42.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_max_scrubs=3 2024-08-19T11:36:42.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:36:42.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:36:42.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=08fb4ddb-a5f4-4f97-be49-4b731ce0d46c 2024-08-19T11:36:42.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 08fb4ddb-a5f4-4f97-be49-4b731ce0d46c' 2024-08-19T11:36:42.448 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 08fb4ddb-a5f4-4f97-be49-4b731ce0d46c 2024-08-19T11:36:42.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:36:42.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKLsNmvrm1GxAAwIstj2xWlsM6PaC63a0CsA== 2024-08-19T11:36:42.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKLsNmvrm1GxAAwIstj2xWlsM6PaC63a0CsA=="}' 2024-08-19T11:36:42.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 08fb4ddb-a5f4-4f97-be49-4b731ce0d46c -i td/ceph-helpers/0/new.json 2024-08-19T11:36:42.774 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:36:42.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:36:42.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQBKLsNmvrm1GxAAwIstj2xWlsM6PaC63a0CsA== --osd-uuid 08fb4ddb-a5f4-4f97-be49-4b731ce0d46c 2024-08-19T11:36:42.825 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:42.823+0000 7f78c711a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:36:42.851 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:42.847+0000 7f78c711a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:36:42.853 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:42.851+0000 7f78c711a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:36:42.853 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:42.851+0000 7f78c711a7c0 -1 bdev(0x55c633b43c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:36:42.853 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:42.851+0000 7f78c711a7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:36:49.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:36:49.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:36:49.370 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:36:49.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:36:49.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:36:49.935 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:36:49.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:36:49.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2024-08-19T11:36:49.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:36:49.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:36:49.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:36:49.967 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:49.963+0000 7fbcab2487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:36:50.005 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:50.003+0000 7fbcab2487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:36:50.007 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:50.003+0000 7fbcab2487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:36:50.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:36:50.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:36:50.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:36:50.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:36:50.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:36:50.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:50.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:36:50.405 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:36:50.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:50.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:50.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:51.030 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:51.027+0000 7fbcab2487c0 -1 Falling back to public interface 2024-08-19T11:36:51.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:51.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:51.832 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:36:51.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:36:51.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:51.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:52.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:53.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:53.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:53.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:36:53.280 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:36:53.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:53.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:53.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:53.939 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:36:53.935+0000 7fbcab2487c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:36:54.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:54.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:54.728 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:36:54.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:36:54.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:54.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:55.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:56.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:56.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:56.216 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:36:56.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:36:56.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:56.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:56.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:57.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:57.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:57.690 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:36:57.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:36:57.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:57.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:58.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:36:59.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:36:59.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:36:59.246 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:36:59.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:36:59.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:36:59.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:36:59.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:00.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:00.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:00.681 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:37:00.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:37:00.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:00.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:01.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:02.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:02.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:02.270 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:37:02.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:37:02.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:02.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:02.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:03.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:03.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:03.721 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:37:03.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:37:03.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:03.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:04.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:05.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:05.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:05.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:37:05.233 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:37:05.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:05.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:05.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:06.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:06.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:06.689 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:37:06.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:37:06.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:06.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:07.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:08.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:08.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:08.303 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:37:08.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:37:08.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:08.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:09.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:10.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:10.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:10.202 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:37:10.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:37:10.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:10.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:11.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:12.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:12.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:12.087 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:37:12.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:37:12.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:12.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:12.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:13.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:13.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:13.943 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:37:13.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:37:13.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:13.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:14.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:15.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:15.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:15.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:37:15.727 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:37:15.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:15.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:17.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:18.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:18.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:18.084 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:37:18.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:37:18.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:18.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:18.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:19.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:19.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:19.526 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:37:19.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:37:19.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:19.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:20.021 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3412261712,v1:127.0.0.1:6803/3412261712] [v2:127.0.0.1:6804/3412261712,v1:127.0.0.1:6805/3412261712] exists,up 08fb4ddb-a5f4-4f97-be49-4b731ce0d46c 2024-08-19T11:37:20.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:37:20.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:37:20.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:37:20.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: get_config osd 0 osd_max_scrubs 2024-08-19T11:37:20.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-08-19T11:37:20.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-08-19T11:37:20.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_max_scrubs 2024-08-19T11:37:20.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_max_scrubs 2024-08-19T11:37:20.024 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-08-19T11:37:20.024 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:37:20.025 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:37:20.025 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:37:20.025 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:20.025 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:20.025 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-osd.0.asok 2024-08-19T11:37:20.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:37:20.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-osd.0.asok config get osd_max_scrubs 2024-08-19T11:37:20.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: test 3 = 3 2024-08-19T11:37:20.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1145: test_get_config: teardown td/ceph-helpers 2024-08-19T11:37:20.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:37:20.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:37:20.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:37:20.133 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:37:20.133 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:37:20.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:37:20.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:37:20.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:37:20.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:37:20.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:37:20.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:37:20.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:37:20.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:37:20.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:37:20.243 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:37:20.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:37:20.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:37:20.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:37:20.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:37:20.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:37:20.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:37:20.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:37:20.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:37:20.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:20.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:20.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:37:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:37:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:37:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:37:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2024-08-19T11:37:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2024-08-19T11:37:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: test_get_is_making_recovery_progress: setup td/ceph-helpers 2024-08-19T11:37:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:37:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:37:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:37:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:37:20.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:37:20.254 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:37:20.254 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:37:20.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:37:20.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:37:20.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:37:20.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:37:20.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:37:20.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:37:20.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:37:20.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:37:20.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:37:20.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:37:20.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:37:20.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:37:20.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:37:20.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:37:20.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:37:20.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:37:20.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:37:20.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:37:20.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:20.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:20.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:37:20.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:37:20.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:37:20.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:37:20.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:37:20.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:20.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:20.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:37:20.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:37:20.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:37:20.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:37:20.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:37:20.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1347: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2024-08-19T11:37:20.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:37:20.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:37:20.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:37:20.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:37:20.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:37:20.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-08-19T11:37:20.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:37:20.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:37:20.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:37:20.749 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:37:20.749 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:20.749 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:20.750 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:37:20.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:37:20.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:37:20.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:37:20.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:37:20.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:37:20.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:37:20.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:37:20.970 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:37:20.970 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:37:20.970 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:37:20.971 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:37:20.971 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:20.971 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:20.971 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:37:20.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:37:20.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:37:21.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:37:21.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:37:21.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:37:21.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:37:21.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:37:21.084 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:37:21.084 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:37:21.084 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:37:21.085 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:37:21.085 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:21.085 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:21.085 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:37:21.085 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:37:21.085 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:37:21.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1348: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2024-08-19T11:37:21.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:37:21.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:37:21.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:37:21.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:37:21.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:37:21.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:37:22.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:37:22.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:37:22.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:37:22.090 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:37:22.090 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:22.090 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:22.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:37:22.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:37:22.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:37:22.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1349: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2024-08-19T11:37:22.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-19T11:37:22.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-19T11:37:22.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-19T11:37:22.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-19T11:37:22.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-19T11:37:22.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-19T11:37:22.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-19T11:37:22.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-19T11:37:22.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1350: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2024-08-19T11:37:22.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:37:22.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:37:22.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:37:22.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:37:22.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:37:22.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:37:22.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:37:22.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:37:22.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:37:22.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:37:22.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:37:22.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:37:22.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:37:22.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:37:22.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:37:22.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:37:22.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:37:22.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:37:22.469 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:37:22.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:37:22.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:37:22.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:37:22.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:37:22.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:22.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:22.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:37:22.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:37:22.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:37:22.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:37:22.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2024-08-19T11:37:22.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2024-08-19T11:37:22.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1535: test_get_last_scrub_stamp: setup td/ceph-helpers 2024-08-19T11:37:22.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:37:22.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:37:22.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:37:22.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:37:22.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:37:22.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:37:22.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:37:22.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:37:22.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:37:22.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:37:22.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:37:22.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:37:22.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:37:22.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:37:22.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:37:22.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:37:22.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:37:22.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:37:22.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:37:22.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:37:22.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:37:22.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:37:22.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:37:22.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:37:22.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:37:22.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:22.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:22.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:37:22.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:37:22.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:37:22.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:37:22.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:37:22.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:22.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:22.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:37:22.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:37:22.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:37:22.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:37:22.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:37:22.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1536: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:37:22.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:37:22.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:37:22.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:37:22.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:37:22.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:37:22.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:37:22.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:37:22.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:37:22.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:37:22.973 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:37:22.973 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:22.973 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:22.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:37:22.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:37:23.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:37:23.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:37:23.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:37:23.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:37:23.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:37:23.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:37:23.220 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:37:23.220 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:37:23.220 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:37:23.221 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:37:23.221 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:23.221 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:23.221 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:37:23.221 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:37:23.221 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:37:23.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:37:23.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:37:23.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:37:23.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:37:23.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:37:23.333 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:37:23.333 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:37:23.333 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:37:23.333 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:37:23.334 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:23.334 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:23.334 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:37:23.334 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:37:23.334 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:37:23.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1537: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2024-08-19T11:37:23.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:37:23.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:37:23.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:37:23.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:37:23.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:37:23.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:37:24.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:37:24.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:37:24.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:37:24.338 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:37:24.338 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:24.338 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:24.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:37:24.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:37:24.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:37:24.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1538: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2024-08-19T11:37:24.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:37:24.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:37:24.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:37:24.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:37:24.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:37:24.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:37:24.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:37:24.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:37:24.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:37:24.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:37:24.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:37:24.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:37:24.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:37:24.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:37:24.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:37:24.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:37:24.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:37:24.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:37:24.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:37:24.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:37:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:37:24.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:37:24.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e85c8f92-50f6-4476-93a2-086650b376a3 2024-08-19T11:37:24.381 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 e85c8f92-50f6-4476-93a2-086650b376a3 2024-08-19T11:37:24.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e85c8f92-50f6-4476-93a2-086650b376a3' 2024-08-19T11:37:24.382 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:37:24.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0LsNmFgfBFxAAiENiCNztq8zg+oXQNEkKTQ== 2024-08-19T11:37:24.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0LsNmFgfBFxAAiENiCNztq8zg+oXQNEkKTQ=="}' 2024-08-19T11:37:24.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e85c8f92-50f6-4476-93a2-086650b376a3 -i td/ceph-helpers/0/new.json 2024-08-19T11:37:24.678 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:37:24.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:37:24.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0LsNmFgfBFxAAiENiCNztq8zg+oXQNEkKTQ== --osd-uuid e85c8f92-50f6-4476-93a2-086650b376a3 2024-08-19T11:37:24.721 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:37:24.718+0000 7fdbc5ea57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:37:24.738 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:37:24.734+0000 7fdbc5ea57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:37:24.740 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:37:24.738+0000 7fdbc5ea57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:37:24.740 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:37:24.738+0000 7fdbc5ea57c0 -1 bdev(0x560f8db16800 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:37:24.740 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:37:24.738+0000 7fdbc5ea57c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:37:31.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:37:31.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:37:31.220 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:37:31.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:37:31.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:37:31.811 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:37:31.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:37:31.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:37:31.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:37:31.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:37:31.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:37:31.843 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:37:31.838+0000 7f0a98c9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:37:31.869 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:37:31.866+0000 7f0a98c9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:37:31.871 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:37:31.866+0000 7f0a98c9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:37:32.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:37:32.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:37:32.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:37:32.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:37:32.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:37:32.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:32.318 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:37:32.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:37:32.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:32.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:32.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:33.391 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:37:33.386+0000 7f0a98c9e7c0 -1 Falling back to public interface 2024-08-19T11:37:33.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:33.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:33.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:37:33.780 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:37:33.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:33.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:34.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:35.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:35.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:35.213 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:37:35.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:37:35.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:35.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:35.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:36.351 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:37:36.346+0000 7f0a98c9e7c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:37:36.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:36.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:36.638 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:37:36.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:37:36.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:36.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:37.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:38.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:38.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:38.233 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:37:38.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:37:38.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:38.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:38.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:39.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:39.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:39.693 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:37:39.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:37:39.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:39.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:40.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:41.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:41.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:41.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:37:41.272 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:37:41.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:41.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:41.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:42.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:42.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:42.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:37:42.724 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:37:42.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:42.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:43.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:44.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:44.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:44.259 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:37:44.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:37:44.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:44.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:44.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:45.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:45.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:45.729 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:37:45.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:37:45.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:45.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:46.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:47.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:47.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:47.265 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:37:47.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:37:47.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:47.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:47.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:48.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:48.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:48.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:37:48.767 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:37:48.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:48.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:49.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:50.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:50.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:50.360 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:37:50.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:37:50.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:50.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:51.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:52.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:52.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:52.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:37:52.227 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:37:52.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:52.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:53.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:54.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:54.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:54.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:37:54.187 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:37:54.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:54.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:55.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:56.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:56.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:56.111 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:37:56.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:37:56.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:56.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:57.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:37:58.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:37:58.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:37:58.630 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:37:58.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:37:58.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:37:58.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:37:59.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:00.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:00.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:00.415 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:38:00.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:38:00.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:00.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:01.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:02.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:02.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:02.430 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:38:02.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:38:02.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:02.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:02.872 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3967549786,v1:127.0.0.1:6803/3967549786] [v2:127.0.0.1:6804/3967549786,v1:127.0.0.1:6805/3967549786] exists,up e85c8f92-50f6-4476-93a2-086650b376a3 2024-08-19T11:38:02.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:38:02.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:38:02.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:38:02.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1539: test_get_last_scrub_stamp: create_rbd_pool 2024-08-19T11:38:02.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:38:03.340 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:38:03.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:38:03.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:38:03.903 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:38:03.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:38:04.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:38:06.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1540: test_get_last_scrub_stamp: wait_for_clean 2024-08-19T11:38:06.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:38:06.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:38:06.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:38:06.048 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:38:06.049 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:38:06.049 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:38:06.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:38:06.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:38:06.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:38:06.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:38:06.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:38:06.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:38:06.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:38:06.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:38:06.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:38:06.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:38:06.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:38:06.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:38:06.620 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:38:06.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836487 2024-08-19T11:38:06.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836487 2024-08-19T11:38:06.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836487' 2024-08-19T11:38:06.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:38:06.785 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-08-19T11:38:06.785 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:38:06.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:38:06.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836487 2024-08-19T11:38:06.788 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:38:06.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836487 2024-08-19T11:38:06.789 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836487 2024-08-19T11:38:06.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-08-19T11:38:06.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:38:07.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836485 -lt 21474836487 2024-08-19T11:38:07.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:38:08.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:38:08.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:38:08.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836487 -lt 21474836487 2024-08-19T11:38:08.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:38:08.631 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:38:08.631 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:38:09.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:38:09.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:38:09.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:38:09.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:38:09.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:38:09.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:38:09.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:38:09.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:38:09.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:38:09.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:38:09.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:38:09.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:38:10.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:38:10.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:38:10.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:38:10.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2024-08-19T11:38:10.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:38:10.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:38:10.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:38:10.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:38:10.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: stamp=2024-08-19T11:38:03.736296+0000 2024-08-19T11:38:10.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1542: test_get_last_scrub_stamp: test -n 2024-08-19T11:38:03.736296+0000 2024-08-19T11:38:10.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1543: test_get_last_scrub_stamp: teardown td/ceph-helpers 2024-08-19T11:38:10.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:38:10.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:38:10.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:38:10.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:38:10.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:38:10.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:38:10.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:38:10.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:38:11.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:38:11.085 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:38:11.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:38:11.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:38:11.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:38:11.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:38:11.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:38:11.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:38:11.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:38:11.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:38:11.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:38:11.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:38:11.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:38:11.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:38:11.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:38:11.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:38:11.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:38:11.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:38:11.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:38:11.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:38:11.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:38:11.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_get_not_primary td/ceph-helpers 2024-08-19T11:38:11.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local dir=td/ceph-helpers 2024-08-19T11:38:11.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: setup td/ceph-helpers 2024-08-19T11:38:11.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:38:11.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:38:11.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:38:11.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:38:11.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:38:11.105 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:38:11.105 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:38:11.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:38:11.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:38:11.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:38:11.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:38:11.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:38:11.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:38:11.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:38:11.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:38:11.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:38:11.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:38:11.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:38:11.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:38:11.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:38:11.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:38:11.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:38:11.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:38:11.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:38:11.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:38:11.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:38:11.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:38:11.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:38:11.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:38:11.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:38:11.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:38:11.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:38:11.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:38:11.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:38:11.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:38:11.119 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:38:11.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:38:11.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:38:11.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:38:11.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-08-19T11:38:11.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:38:11.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:38:11.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:38:11.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:38:11.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:38:11.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-08-19T11:38:11.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:38:11.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:38:11.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:38:11.634 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:38:11.634 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:38:11.634 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:38:11.634 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:38:11.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-08-19T11:38:11.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:38:11.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:38:11.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:38:11.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:38:11.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:38:11.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:38:11.887 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:38:11.887 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:38:11.887 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:38:11.888 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:38:11.888 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:38:11.889 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:38:11.889 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:38:11.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:38:11.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:38:12.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:38:12.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:38:12.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:38:12.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:38:12.004 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:38:12.004 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:38:12.004 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:38:12.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:38:12.005 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:38:12.005 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:38:12.005 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:38:12.005 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:38:12.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:38:12.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:38:12.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: run_mgr td/ceph-helpers x 2024-08-19T11:38:12.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:38:12.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:38:12.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:38:12.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:38:12.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:38:12.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:38:13.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:38:13.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:38:13.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:38:13.015 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:38:13.015 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:38:13.015 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:38:13.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:38:13.016 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:38:13.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:38:13.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1243: test_get_not_primary: run_osd td/ceph-helpers 0 2024-08-19T11:38:13.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:38:13.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:38:13.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:38:13.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:38:13.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:38:13.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:38:13.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:38:13.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:38:13.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:38:13.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:38:13.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:38:13.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:38:13.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:38:13.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:38:13.057 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:38:13.057 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:38:13.057 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:38:13.057 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:38:13.058 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:38:13.058 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:38:13.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:38:13.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:38:13.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:38:13.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:38:13.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:38:13.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:38:13.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:38:13.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:38:13.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:38:13.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:38:13.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:38:13.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:38:13.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:38:13.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:38:13.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:38:13.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a3d3580-e711-4b84-8e6a-88b428ad6464 2024-08-19T11:38:13.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0a3d3580-e711-4b84-8e6a-88b428ad6464' 2024-08-19T11:38:13.060 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 0a3d3580-e711-4b84-8e6a-88b428ad6464 2024-08-19T11:38:13.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:38:13.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQClLsNmexCaBBAASlnpmbL8hHTP3hfgRe5iVA== 2024-08-19T11:38:13.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQClLsNmexCaBBAASlnpmbL8hHTP3hfgRe5iVA=="}' 2024-08-19T11:38:13.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a3d3580-e711-4b84-8e6a-88b428ad6464 -i td/ceph-helpers/0/new.json 2024-08-19T11:38:13.364 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:38:13.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:38:13.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQClLsNmexCaBBAASlnpmbL8hHTP3hfgRe5iVA== --osd-uuid 0a3d3580-e711-4b84-8e6a-88b428ad6464 2024-08-19T11:38:13.410 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:13.410+0000 7fc8fecca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:38:13.432 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:13.434+0000 7fc8fecca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:38:13.433 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:13.434+0000 7fc8fecca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:38:13.434 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:13.434+0000 7fc8fecca7c0 -1 bdev(0x5570e2be1c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:38:13.434 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:13.434+0000 7fc8fecca7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:38:20.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:38:20.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:38:20.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:38:20.571 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:38:20.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:38:21.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:38:21.144 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:38:21.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:38:21.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:38:21.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:38:21.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:38:21.175 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:21.173+0000 7fc1485a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:38:21.219 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:21.217+0000 7fc1485a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:38:21.220 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:21.221+0000 7fc1485a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:38:21.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:38:21.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:38:21.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:38:21.561 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:38:21.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:38:21.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:38:21.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:21.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:38:21.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:21.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:21.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:22.747 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:22.745+0000 7fc1485a67c0 -1 Falling back to public interface 2024-08-19T11:38:22.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:22.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:22.987 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:38:22.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:38:22.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:22.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:23.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:24.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:24.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:24.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:38:24.405 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:38:24.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:24.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:24.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:25.569 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:25.569+0000 7fc1485a67c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:38:25.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:25.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:25.845 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:38:25.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:38:25.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:25.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:26.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:27.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:27.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:27.363 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:38:27.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:38:27.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:27.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:27.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:28.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:28.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:28.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:38:28.843 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:38:28.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:28.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:29.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:30.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:30.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:30.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:38:30.435 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:38:30.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:30.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:30.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:31.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:31.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:31.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:38:31.901 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:38:31.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:31.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:32.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:38:33.419 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:38:33.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:33.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:33.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:34.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:34.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:34.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:38:34.866 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:38:34.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:34.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:35.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:36.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:36.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:36.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:38:36.380 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:38:36.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:36.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:36.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:37.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:37.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:37.845 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:38:37.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:38:37.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:37.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:38.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:39.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:39.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:39.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:38:39.413 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:38:39.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:39.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:40.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:41.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:41.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:41.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:38:41.250 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:38:41.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:41.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:42.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:43.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:43.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:43.196 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:38:43.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:38:43.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:43.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:44.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:45.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:45.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:45.020 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:38:45.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:38:45.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:45.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:45.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:46.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:46.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:46.926 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:38:46.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:38:46.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:46.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:48.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:49.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:49.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:49.239 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:38:49.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:38:49.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:49.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:50.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:38:51.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:38:51.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:38:51.189 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:38:51.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:38:51.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:38:51.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:38:51.646 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1420997714,v1:127.0.0.1:6803/1420997714] [v2:127.0.0.1:6804/1420997714,v1:127.0.0.1:6805/1420997714] exists,up 0a3d3580-e711-4b84-8e6a-88b428ad6464 2024-08-19T11:38:51.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:38:51.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:38:51.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:38:51.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: test_get_not_primary: run_osd td/ceph-helpers 1 2024-08-19T11:38:51.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:38:51.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:38:51.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T11:38:51.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:38:51.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-08-19T11:38:51.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:38:51.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:38:51.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:38:51.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:38:51.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-08-19T11:38:51.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-08-19T11:38:51.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:38:51.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:38:51.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:38:51.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:38:51.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:38:51.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:38:51.651 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:38:51.651 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:38:51.651 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:38:51.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:38:51.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:38:51.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:38:51.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:38:51.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:38:51.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:38:51.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:38:51.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:38:51.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:38:51.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:38:51.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:38:51.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:38:51.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:38:51.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-08-19T11:38:51.653 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:38:51.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a4e8b98d-c030-4d48-a797-8578536387ef 2024-08-19T11:38:51.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a4e8b98d-c030-4d48-a797-8578536387ef' 2024-08-19T11:38:51.653 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 a4e8b98d-c030-4d48-a797-8578536387ef 2024-08-19T11:38:51.653 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:38:51.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLLsNmB4PtJxAAof2llBQdBFEMmpSoSGX6/Q== 2024-08-19T11:38:51.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLLsNmB4PtJxAAof2llBQdBFEMmpSoSGX6/Q=="}' 2024-08-19T11:38:51.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a4e8b98d-c030-4d48-a797-8578536387ef -i td/ceph-helpers/1/new.json 2024-08-19T11:38:52.259 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:38:52.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-08-19T11:38:52.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLLsNmB4PtJxAAof2llBQdBFEMmpSoSGX6/Q== --osd-uuid a4e8b98d-c030-4d48-a797-8578536387ef 2024-08-19T11:38:52.309 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:52.309+0000 7fa21316e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:38:52.331 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:52.329+0000 7fa21316e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:38:52.332 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:52.333+0000 7fa21316e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:38:52.332 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:52.333+0000 7fa21316e7c0 -1 bdev(0x56399fa28800 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-08-19T11:38:52.332 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:38:52.333+0000 7fa21316e7c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-08-19T11:38:59.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-08-19T11:38:59.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:38:59.494 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-08-19T11:38:59.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T11:38:59.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:39:00.156 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-08-19T11:39:00.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T11:39:00.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:39:00.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:39:00.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:39:00.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:39:00.189 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:00.189+0000 7ff8944b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:39:00.218 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:00.217+0000 7ff8944b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:39:00.219 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:00.221+0000 7ff8944b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:39:00.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T11:39:00.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:39:00.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T11:39:00.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:39:00.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:39:00.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:00.568 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:39:00.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:39:00.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:00.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:01.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:01.438 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:01.437+0000 7ff8944b77c0 -1 Falling back to public interface 2024-08-19T11:39:02.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:02.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:02.035 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:39:02.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:39:02.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:02.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:02.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:03.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:03.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:03.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:39:03.471 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:39:03.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:03.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:03.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:04.393 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:04.393+0000 7ff8944b77c0 -1 osd.1 0 log_to_monitors true 2024-08-19T11:39:04.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:04.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:04.956 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:39:04.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:39:04.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:04.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:05.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:06.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:06.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:06.577 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:39:06.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:39:06.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:06.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:07.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:08.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:08.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:08.013 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:39:08.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:39:08.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:08.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:08.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:09.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:09.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:09.716 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:39:09.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:39:09.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:09.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:10.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:11.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:11.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:11.165 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:39:11.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:39:11.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:11.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:11.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:12.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:12.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:12.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:39:12.658 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:39:12.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:12.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:13.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:14.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:14.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:14.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:39:14.116 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:39:14.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:14.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:14.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:15.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:15.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:15.696 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:39:15.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:39:15.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:15.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:16.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:17.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:17.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:17.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:39:17.334 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:39:17.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:17.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:17.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:18.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:18.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:18.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:39:18.784 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:39:18.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:18.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:19.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:20.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:20.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:20.236 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:39:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:39:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:21.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:39:22.188 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:39:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:23.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:24.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:24.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:24.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:39:24.158 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:39:24.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:24.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:24.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:25.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:25.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:25.985 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:39:25.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:39:25.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:25.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:26.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:27.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:27.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:27.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:39:27.921 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:39:27.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:27.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:28.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:29.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:29.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:29.392 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:39:29.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:39:29.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:29.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:30.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:31.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:31.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:31.226 INFO:tasks.workunit.client.0.smithi130.stdout:19 2024-08-19T11:39:31.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T11:39:31.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:31.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:39:31.718 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/330565324,v1:127.0.0.1:6811/330565324] [v2:127.0.0.1:6812/330565324,v1:127.0.0.1:6813/330565324] exists,up a4e8b98d-c030-4d48-a797-8578536387ef 2024-08-19T11:39:31.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:39:31.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:39:31.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:39:31.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: test_get_not_primary: create_rbd_pool 2024-08-19T11:39:31.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:39:32.129 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:39:32.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:39:32.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:39:32.674 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:39:32.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:39:33.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:39:36.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1246: test_get_not_primary: wait_for_clean 2024-08-19T11:39:36.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:39:36.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:39:36.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:39:36.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:39:36.178 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:39:36.179 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:39:36.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:39:36.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:39:36.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:39:36.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:39:36.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:39:36.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:39:36.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:39:36.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:39:36.301 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:39:36.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-19T11:39:36.738 INFO:tasks.workunit.client.0.smithi130.stderr:1' 2024-08-19T11:39:36.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:39:36.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:39:36.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:39:36.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836494 2024-08-19T11:39:36.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836494 2024-08-19T11:39:36.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836494' 2024-08-19T11:39:36.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:39:36.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T11:39:37.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672970 2024-08-19T11:39:37.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672970 2024-08-19T11:39:37.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836494 1-42949672970' 2024-08-19T11:39:37.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:39:37.073 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836494 2024-08-19T11:39:37.073 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:39:37.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:39:37.075 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836494 2024-08-19T11:39:37.075 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:39:37.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836494 2024-08-19T11:39:37.076 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836494 2024-08-19T11:39:37.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-08-19T11:39:37.076 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:39:37.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836492 -lt 21474836494 2024-08-19T11:39:37.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:39:38.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:39:38.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:39:38.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836495 -lt 21474836494 2024-08-19T11:39:38.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:39:38.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672970 2024-08-19T11:39:38.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:39:38.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-19T11:39:38.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672970 2024-08-19T11:39:38.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:39:38.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672970 2024-08-19T11:39:38.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-08-19T11:39:38.909 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672970 2024-08-19T11:39:38.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T11:39:39.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672970 -lt 42949672970 2024-08-19T11:39:39.359 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:39:39.360 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:39:39.360 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:39:39.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:39:39.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:39:39.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:39:39.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:39:39.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:39:39.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:39:39.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:39:39.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:39:40.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:39:40.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:39:40.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:39:40.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:39:40.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:39:40.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:39:40.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:39:40.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: get_primary rbd GROUP 2024-08-19T11:39:40.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-08-19T11:39:40.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-08-19T11:39:40.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T11:39:40.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-08-19T11:39:41.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: local primary=1 2024-08-19T11:39:41.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: get_not_primary rbd GROUP 2024-08-19T11:39:41.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-08-19T11:39:41.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=GROUP 2024-08-19T11:39:41.387 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd GROUP 2024-08-19T11:39:41.387 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-08-19T11:39:41.388 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-08-19T11:39:41.388 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T11:39:41.388 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-08-19T11:39:41.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-08-19T11:39:41.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd GROUP 2024-08-19T11:39:41.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-08-19T11:39:42.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: local not_primary=0 2024-08-19T11:39:42.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: test_get_not_primary: test 0 '!=' 1 2024-08-19T11:39:42.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: test_get_not_primary: test 0 = 0 -o 0 = 1 2024-08-19T11:39:42.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: test_get_not_primary: teardown td/ceph-helpers 2024-08-19T11:39:42.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:39:42.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:39:42.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:39:42.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:39:42.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:39:42.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:39:42.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:39:42.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:39:42.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:39:42.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:39:42.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:39:42.430 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:39:42.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:39:42.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:39:42.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:39:42.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:39:42.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:39:42.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:39:42.433 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:39:42.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:39:42.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:39:42.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:39:42.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:39:42.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:39:42.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:39:42.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:39:42.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:39:42.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:39:42.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:39:42.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_get_num_active_clean td/ceph-helpers 2024-08-19T11:39:42.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: local dir=td/ceph-helpers 2024-08-19T11:39:42.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1374: test_get_num_active_clean: setup td/ceph-helpers 2024-08-19T11:39:42.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:39:42.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:39:42.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:39:42.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:39:42.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:39:42.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:39:42.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:39:42.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:39:42.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:39:42.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:39:42.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:39:42.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:39:42.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:39:42.453 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:39:42.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:39:42.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:39:42.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:39:42.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:39:42.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:39:42.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:39:42.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:39:42.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:39:42.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:39:42.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:39:42.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:39:42.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:39:42.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:39:42.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:39:42.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:39:42.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:39:42.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:39:42.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:39:42.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:39:42.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:39:42.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:39:42.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:39:42.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:39:42.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:39:42.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:39:42.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1375: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:39:42.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:39:42.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:39:42.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:39:42.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:39:42.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:39:42.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:39:42.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:39:42.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:39:42.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:39:42.920 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:39:42.920 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:39:42.920 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:39:42.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:39:42.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:39:43.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:39:43.157 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:39:43.158 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:39:43.158 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:39:43.158 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:39:43.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:39:43.159 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:39:43.159 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:39:43.159 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:39:43.160 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:39:43.160 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:39:43.160 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:39:43.160 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:39:43.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:39:43.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:39:43.271 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:39:43.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:39:43.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:39:43.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:39:43.273 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:39:43.273 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:39:43.273 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:39:43.273 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:39:43.273 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:39:43.273 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:39:43.273 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:39:43.274 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:39:43.274 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:39:43.274 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:39:43.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1376: test_get_num_active_clean: run_mgr td/ceph-helpers x 2024-08-19T11:39:43.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:39:43.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:39:43.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:39:43.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:39:43.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:39:43.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:39:44.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:39:44.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:39:44.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:39:44.279 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:39:44.279 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:39:44.279 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:39:44.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:39:44.280 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:39:44.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:39:44.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1377: test_get_num_active_clean: run_osd td/ceph-helpers 0 2024-08-19T11:39:44.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:39:44.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:39:44.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:39:44.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:39:44.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:39:44.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:39:44.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:39:44.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:39:44.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:39:44.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:39:44.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:39:44.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:39:44.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:39:44.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:39:44.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:39:44.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:39:44.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:39:44.316 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:39:44.316 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:39:44.316 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:39:44.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:39:44.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:39:44.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:39:44.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:39:44.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:39:44.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:39:44.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:39:44.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:39:44.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:39:44.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:39:44.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:39:44.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:39:44.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:39:44.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:39:44.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:39:44.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c66bc280-31f3-4120-8f85-41cfd1bd1884 2024-08-19T11:39:44.318 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 c66bc280-31f3-4120-8f85-41cfd1bd1884 2024-08-19T11:39:44.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c66bc280-31f3-4120-8f85-41cfd1bd1884' 2024-08-19T11:39:44.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:39:44.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAAL8NmCWMYFBAA1xibJ5pt+PAHJ8ot/U++tw== 2024-08-19T11:39:44.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAAL8NmCWMYFBAA1xibJ5pt+PAHJ8ot/U++tw=="}' 2024-08-19T11:39:44.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c66bc280-31f3-4120-8f85-41cfd1bd1884 -i td/ceph-helpers/0/new.json 2024-08-19T11:39:44.651 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:39:44.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:39:44.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAAL8NmCWMYFBAA1xibJ5pt+PAHJ8ot/U++tw== --osd-uuid c66bc280-31f3-4120-8f85-41cfd1bd1884 2024-08-19T11:39:44.697 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:44.696+0000 7f0b891be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:39:44.728 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:44.728+0000 7f0b891be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:39:44.729 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:44.728+0000 7f0b891be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:39:44.730 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:44.728+0000 7f0b891be7c0 -1 bdev(0x5648ee421c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:39:44.730 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:44.728+0000 7f0b891be7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:39:51.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:39:51.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:39:51.099 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:39:51.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:39:51.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:39:51.689 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:39:51.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:39:51.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:39:51.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:39:51.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:39:51.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:39:51.721 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:51.720+0000 7fbb580d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:39:51.741 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:51.740+0000 7fbb580d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:39:51.743 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:51.740+0000 7fbb580d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:39:52.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:39:52.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:39:52.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:39:52.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:39:52.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:39:52.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:52.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:39:52.181 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:39:52.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:52.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:39:52.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:53.018 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:53.016+0000 7fbb580d47c0 -1 Falling back to public interface 2024-08-19T11:39:53.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:53.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:53.621 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:39:53.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:39:53.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:53.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:39:54.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:55.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:55.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:55.046 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:39:55.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:39:55.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:55.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:39:55.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:55.975 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:39:55.972+0000 7fbb580d47c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:39:56.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:56.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:56.476 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:39:56.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:39:56.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:56.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:39:56.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:57.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:57.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:57.997 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:39:57.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:39:57.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:57.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:39:58.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:39:59.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:39:59.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:39:59.425 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:39:59.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:39:59.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:39:59.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:39:59.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:00.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:00.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:00.923 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:40:00.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:40:00.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:00.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:01.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:02.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:02.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:02.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:40:02.387 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:40:02.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:02.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:02.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:03.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:03.963 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:40:03.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:03.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:40:03.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:03.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:04.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:05.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:05.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:05.417 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:40:05.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:40:05.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:05.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:05.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:06.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:06.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:06.945 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:40:06.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:40:06.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:06.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:07.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:08.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:08.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:08.426 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:40:08.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:40:08.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:08.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:08.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:09.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:09.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:09.980 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:40:09.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:40:09.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:09.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:10.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:11.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:11.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:11.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:40:11.652 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:40:11.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:11.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:12.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:13.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:13.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:13.805 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:40:13.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:40:13.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:13.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:14.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:15.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:15.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:15.636 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:40:15.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:40:15.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:15.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:16.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:17.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:17.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:17.444 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:40:17.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:40:17.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:17.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:18.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:19.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:19.412 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:40:19.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:19.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:40:19.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:19.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:20.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:21.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:21.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:21.036 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:40:21.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:40:21.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:21.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:21.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:22.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:22.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:22.666 INFO:tasks.workunit.client.0.smithi130.stdout:19 2024-08-19T11:40:22.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T11:40:22.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:22.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:23.111 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2824660318,v1:127.0.0.1:6803/2824660318] [v2:127.0.0.1:6804/2824660318,v1:127.0.0.1:6805/2824660318] exists,up c66bc280-31f3-4120-8f85-41cfd1bd1884 2024-08-19T11:40:23.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:40:23.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:40:23.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:40:23.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1378: test_get_num_active_clean: create_rbd_pool 2024-08-19T11:40:23.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:40:23.614 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:40:23.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:40:23.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:40:24.266 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:40:24.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:40:25.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:40:26.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1379: test_get_num_active_clean: wait_for_clean 2024-08-19T11:40:26.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:40:26.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:40:26.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:40:26.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:40:26.574 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:40:26.574 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:40:26.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:40:26.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:40:26.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:40:26.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:40:26.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:40:26.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:40:26.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:40:26.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:40:26.693 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:40:27.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:40:27.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:40:27.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:40:27.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:40:27.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836486 2024-08-19T11:40:27.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836486 2024-08-19T11:40:27.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836486' 2024-08-19T11:40:27.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:40:27.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836486 2024-08-19T11:40:27.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:40:27.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:40:27.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836486 2024-08-19T11:40:27.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:40:27.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836486 2024-08-19T11:40:27.341 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836486 2024-08-19T11:40:27.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-08-19T11:40:27.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:40:27.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836484 -lt 21474836486 2024-08-19T11:40:27.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:40:28.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:40:28.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:40:29.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836487 -lt 21474836486 2024-08-19T11:40:29.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:40:29.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:40:29.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:40:29.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:40:29.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:40:29.979 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:40:29.979 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:40:29.979 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:40:29.979 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:40:29.979 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:40:29.979 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:40:30.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:40:30.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:40:30.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:40:30.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:40:30.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:40:30.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:40:30.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:40:30.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: get_num_active_clean 2024-08-19T11:40:30.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:40:30.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:40:30.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:40:30.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:40:30.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:40:31.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: local num_active_clean=4 2024-08-19T11:40:31.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1381: test_get_num_active_clean: test 4 = 4 2024-08-19T11:40:31.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1382: test_get_num_active_clean: teardown td/ceph-helpers 2024-08-19T11:40:31.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:40:31.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:40:31.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:40:31.400 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:40:31.400 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:40:31.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:40:31.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:40:31.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:40:31.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:40:31.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:40:31.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:40:31.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:40:31.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:40:31.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:40:31.508 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:40:31.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:40:31.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:40:31.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:40:31.510 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:40:31.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:40:31.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:40:31.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:40:31.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:40:31.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:40:31.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:40:31.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:40:31.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:40:31.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:40:31.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:40:31.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_get_num_active_or_peered td/ceph-helpers 2024-08-19T11:40:31.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: local dir=td/ceph-helpers 2024-08-19T11:40:31.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1404: test_get_num_active_or_peered: setup td/ceph-helpers 2024-08-19T11:40:31.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:40:31.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:40:31.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:40:31.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:40:31.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:40:31.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:40:31.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:40:31.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:40:31.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:40:31.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:40:31.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:40:31.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:40:31.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:40:31.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:40:31.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:40:31.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:40:31.523 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:40:31.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:40:31.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:40:31.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:40:31.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:40:31.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:40:31.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:40:31.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:40:31.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:40:31.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:40:31.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:40:31.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:40:31.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:40:31.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:40:31.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:40:31.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:40:31.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:40:31.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:40:31.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:40:31.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:40:31.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:40:31.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:40:31.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:40:31.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1405: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:40:31.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:40:31.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:40:31.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:40:31.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:40:31.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:40:31.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:40:32.035 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:40:32.035 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:40:32.035 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:40:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:40:32.036 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:40:32.036 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:40:32.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:40:32.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:40:32.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:40:32.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:40:32.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:40:32.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:40:32.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:40:32.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:40:32.263 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:40:32.263 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:40:32.264 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:40:32.264 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:40:32.264 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:40:32.264 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:40:32.264 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:40:32.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:40:32.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:40:32.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:40:32.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:40:32.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:40:32.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:40:32.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:40:32.377 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:40:32.377 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:40:32.377 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:40:32.377 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:40:32.377 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:40:32.377 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:40:32.377 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:40:32.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:40:32.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:40:32.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1406: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2024-08-19T11:40:32.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:40:32.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:40:32.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:40:32.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:40:32.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:40:32.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:40:33.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:40:33.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:40:33.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:40:33.376 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:40:33.376 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:40:33.376 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:40:33.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:40:33.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:40:33.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:40:33.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1407: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2024-08-19T11:40:33.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:40:33.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:40:33.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:40:33.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:40:33.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:40:33.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:40:33.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:40:33.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:40:33.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:40:33.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:40:33.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:40:33.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:40:33.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:40:33.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:40:33.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:40:33.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:40:33.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:40:33.418 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:40:33.418 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:40:33.418 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:40:33.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:40:33.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:40:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:40:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:40:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:40:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:40:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:40:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:40:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:40:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:40:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:40:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:40:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:40:33.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:40:33.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:40:33.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ca5f43b6-5699-4496-b3f9-bff6d1d1b8f8 2024-08-19T11:40:33.420 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 ca5f43b6-5699-4496-b3f9-bff6d1d1b8f8 2024-08-19T11:40:33.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ca5f43b6-5699-4496-b3f9-bff6d1d1b8f8' 2024-08-19T11:40:33.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:40:33.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxL8NmStMaGhAAE/3kYxyUF3EUwAOgMhbhpw== 2024-08-19T11:40:33.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxL8NmStMaGhAAE/3kYxyUF3EUwAOgMhbhpw=="}' 2024-08-19T11:40:33.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ca5f43b6-5699-4496-b3f9-bff6d1d1b8f8 -i td/ceph-helpers/0/new.json 2024-08-19T11:40:33.724 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:40:33.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:40:33.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxL8NmStMaGhAAE/3kYxyUF3EUwAOgMhbhpw== --osd-uuid ca5f43b6-5699-4496-b3f9-bff6d1d1b8f8 2024-08-19T11:40:33.778 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:40:33.775+0000 7f25775f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:40:33.793 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:40:33.791+0000 7f25775f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:40:33.794 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:40:33.791+0000 7f25775f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:40:33.794 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:40:33.791+0000 7f25775f37c0 -1 bdev(0x55e97ed8bc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:40:33.794 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:40:33.791+0000 7f25775f37c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:40:40.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:40:40.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:40:40.340 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:40:40.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:40:40.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:40:40.919 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:40:40.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:40:40.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:40:40.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:40:40.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:40:40.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:40:40.952 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:40:40.947+0000 7f9befe877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:40:40.953 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:40:40.951+0000 7f9befe877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:40:40.955 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:40:40.951+0000 7f9befe877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:40:41.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:40:41.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:40:41.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:40:41.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:40:41.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:40:41.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:41.377 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:40:41.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:40:41.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:41.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:41.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:41.986 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:40:41.983+0000 7f9befe877c0 -1 Falling back to public interface 2024-08-19T11:40:42.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:42.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:42.859 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:40:42.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:40:42.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:42.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:43.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:44.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:44.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:44.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:40:44.341 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:40:44.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:44.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:44.679 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:40:44.675+0000 7f9befe877c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:40:44.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:45.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:45.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:45.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:40:45.766 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:40:45.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:45.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:46.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:47.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:47.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:47.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:40:47.306 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:40:47.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:47.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:47.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:48.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:48.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:48.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:40:48.848 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:40:48.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:48.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:49.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:50.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:50.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:50.278 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:40:50.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:40:50.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:50.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:50.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:51.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:51.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:51.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:40:51.818 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:40:51.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:51.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:52.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:53.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:53.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:53.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:40:53.300 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:40:53.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:53.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:53.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:54.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:54.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:54.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:40:54.776 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:40:54.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:54.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:55.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:56.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:56.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:56.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:40:56.266 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:40:56.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:56.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:56.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:57.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:57.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:40:57.789 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:40:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:40:58.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:40:59.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:40:59.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:40:59.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:40:59.304 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:40:59.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:40:59.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:00.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:01.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:01.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:01.360 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:41:01.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:41:01.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:01.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:02.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:03.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:03.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:03.253 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:41:03.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:41:03.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:03.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:03.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:04.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:04.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:04.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:41:04.877 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:41:04.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:04.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:06.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:07.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:07.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:07.296 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:41:07.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:41:07.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:07.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:08.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:09.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:09.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:09.068 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:41:09.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:41:09.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:09.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:09.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:10.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:10.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:10.987 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:41:10.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:41:10.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:10.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:11.523 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3956854087,v1:127.0.0.1:6803/3956854087] [v2:127.0.0.1:6804/3956854087,v1:127.0.0.1:6805/3956854087] exists,up ca5f43b6-5699-4496-b3f9-bff6d1d1b8f8 2024-08-19T11:41:11.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:41:11.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:41:11.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:41:11.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1408: test_get_num_active_or_peered: create_rbd_pool 2024-08-19T11:41:11.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:41:11.997 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:41:12.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:41:12.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:41:12.592 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:41:12.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:41:13.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:41:14.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1409: test_get_num_active_or_peered: wait_for_clean 2024-08-19T11:41:14.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:41:14.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:41:14.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:41:14.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:41:14.977 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:41:14.977 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:41:14.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:41:14.978 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:41:14.978 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:41:15.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:41:15.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:41:15.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:41:15.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:41:15.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:41:15.099 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:41:15.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:41:15.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:41:15.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:41:15.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:41:15.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836492 2024-08-19T11:41:15.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836492 2024-08-19T11:41:15.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836492' 2024-08-19T11:41:15.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:41:15.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836492 2024-08-19T11:41:15.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:41:15.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:41:15.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836492 2024-08-19T11:41:15.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:41:15.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836492 2024-08-19T11:41:15.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-08-19T11:41:15.722 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836492 2024-08-19T11:41:15.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:41:16.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836490 -lt 21474836492 2024-08-19T11:41:16.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:41:17.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:41:17.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:41:17.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836492 -lt 21474836492 2024-08-19T11:41:17.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:41:17.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:41:17.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:41:18.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:41:18.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:41:18.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:41:18.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:41:18.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:41:18.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:41:18.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:41:18.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:41:18.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:41:18.720 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:41:18.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:41:18.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:41:19.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:41:19.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:41:19.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:41:19.278 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: get_num_active_or_peered 2024-08-19T11:41:19.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-08-19T11:41:19.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-08-19T11:41:19.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-08-19T11:41:19.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-08-19T11:41:19.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-08-19T11:41:19.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: local num_peered=4 2024-08-19T11:41:19.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1411: test_get_num_active_or_peered: test 4 = 4 2024-08-19T11:41:19.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1412: test_get_num_active_or_peered: teardown td/ceph-helpers 2024-08-19T11:41:19.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:41:19.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:41:19.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:41:19.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:41:19.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:41:19.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:41:19.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:41:19.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:41:19.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:41:19.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:41:19.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:41:19.785 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:41:19.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:41:19.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:41:19.786 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:41:19.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:41:19.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:41:19.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:41:19.788 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:41:19.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:41:19.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:41:19.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:41:19.796 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:41:19.796 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:41:19.796 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:41:19.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:41:19.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:41:19.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:41:19.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:41:19.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_get_num_pgs td/ceph-helpers 2024-08-19T11:41:19.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: local dir=td/ceph-helpers 2024-08-19T11:41:19.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1431: test_get_num_pgs: setup td/ceph-helpers 2024-08-19T11:41:19.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:41:19.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:41:19.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:41:19.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:41:19.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:41:19.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:41:19.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:41:19.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:41:19.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:41:19.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:41:19.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:41:19.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:41:19.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:41:19.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:41:19.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:41:19.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:41:19.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:41:19.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:41:19.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:41:19.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:41:19.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:41:19.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:41:19.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:41:19.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:41:19.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:41:19.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:41:19.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:41:19.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:41:19.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:41:19.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:41:19.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:41:19.811 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:41:19.811 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:41:19.811 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:41:19.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:41:19.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:41:19.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:41:19.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:41:19.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:41:19.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:41:19.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:41:19.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:41:19.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:41:19.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:41:19.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:41:19.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:41:20.329 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:41:20.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:41:20.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:41:20.331 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:41:20.331 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:41:20.331 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:41:20.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:41:20.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:41:20.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:41:20.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:41:20.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:41:20.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:41:20.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:41:20.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:41:20.544 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:41:20.545 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:41:20.545 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:41:20.545 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:41:20.545 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:41:20.545 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:41:20.545 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:41:20.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:41:20.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:41:20.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:41:20.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:41:20.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:41:20.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:41:20.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:41:20.659 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:41:20.659 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:41:20.659 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:41:20.659 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:41:20.659 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:41:20.659 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:41:20.660 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:41:20.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:41:20.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:41:20.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: test_get_num_pgs: run_mgr td/ceph-helpers x 2024-08-19T11:41:20.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:41:20.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:41:20.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:41:20.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:41:20.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:41:20.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:41:21.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:41:21.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:41:21.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:41:21.680 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:41:21.680 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:41:21.680 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:41:21.680 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:41:21.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:41:21.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:41:21.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: test_get_num_pgs: run_osd td/ceph-helpers 0 2024-08-19T11:41:21.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:41:21.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:41:21.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:41:21.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:41:21.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:41:21.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:41:21.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:41:21.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:41:21.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:41:21.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:41:21.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:41:21.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:41:21.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:41:21.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:41:21.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:41:21.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:41:21.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:41:21.721 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:41:21.721 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:41:21.721 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:41:21.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:41:21.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:41:21.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:41:21.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:41:21.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:41:21.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:41:21.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:41:21.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:41:21.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:41:21.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:41:21.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:41:21.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:41:21.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:41:21.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:41:21.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:41:21.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3e85ea0b-35f2-48ad-a15a-187736a5768b 2024-08-19T11:41:21.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3e85ea0b-35f2-48ad-a15a-187736a5768b' 2024-08-19T11:41:21.724 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 3e85ea0b-35f2-48ad-a15a-187736a5768b 2024-08-19T11:41:21.724 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:41:21.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhL8Nm9jEsLBAAIRG2ZH5jlFrE+Px1f8onZg== 2024-08-19T11:41:21.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhL8Nm9jEsLBAAIRG2ZH5jlFrE+Px1f8onZg=="}' 2024-08-19T11:41:21.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3e85ea0b-35f2-48ad-a15a-187736a5768b -i td/ceph-helpers/0/new.json 2024-08-19T11:41:22.061 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:41:22.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:41:22.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhL8Nm9jEsLBAAIRG2ZH5jlFrE+Px1f8onZg== --osd-uuid 3e85ea0b-35f2-48ad-a15a-187736a5768b 2024-08-19T11:41:22.116 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:41:22.114+0000 7f04ac3157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:41:22.129 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:41:22.126+0000 7f04ac3157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:41:22.130 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:41:22.126+0000 7f04ac3157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:41:22.131 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:41:22.126+0000 7f04ac3157c0 -1 bdev(0x55c61c965c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:41:22.131 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:41:22.126+0000 7f04ac3157c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:41:28.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:41:28.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:41:28.416 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:41:28.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:41:28.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:41:29.000 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:41:29.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:41:29.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:41:29.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:41:29.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:41:29.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:41:29.034 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:41:29.034+0000 7fd8ced377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:41:29.067 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:41:29.066+0000 7fd8ced377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:41:29.069 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:41:29.070+0000 7fd8ced377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:41:29.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:41:29.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:41:29.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:41:29.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:41:29.449 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:41:29.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:41:29.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:29.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:41:29.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:29.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:29.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:30.086 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:41:30.086+0000 7fd8ced377c0 -1 Falling back to public interface 2024-08-19T11:41:30.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:30.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:30.916 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:41:30.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:41:30.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:30.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:31.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:32.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:32.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:32.336 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:41:32.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:41:32.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:32.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:32.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:32.940 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:41:32.942+0000 7fd8ced377c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:41:33.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:33.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:33.767 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:41:33.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:41:33.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:33.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:34.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:35.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:35.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:35.329 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:41:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:41:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:35.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:36.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:36.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:36.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:41:36.933 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:41:36.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:36.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:37.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:38.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:38.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:38.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:41:38.405 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:41:38.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:38.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:38.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:39.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:39.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:39.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:41:39.950 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:41:39.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:39.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:40.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:41.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:41.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:41.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:41:41.432 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:41:41.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:41.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:41.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:42.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:42.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:42.943 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:41:42.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:41:42.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:42.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:43.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:44.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:44.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:44.367 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:41:44.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:41:44.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:44.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:44.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:45.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:45.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:45.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:41:45.976 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:41:45.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:45.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:46.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:47.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:47.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:47.421 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:41:47.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:41:47.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:47.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:47.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:48.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:48.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:48.864 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:41:48.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:41:48.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:48.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:49.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:50.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:50.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:50.903 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:41:50.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:41:50.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:50.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:51.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:52.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:52.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:52.811 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:41:52.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:41:52.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:52.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:53.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:54.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:54.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:54.951 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:41:54.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:41:54.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:54.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:55.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:56.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:56.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:56.676 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:41:56.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:41:56.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:56.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:57.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:41:58.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:41:58.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:41:58.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:41:58.705 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:41:58.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:41:58.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:41:59.141 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1656381175,v1:127.0.0.1:6803/1656381175] [v2:127.0.0.1:6804/1656381175,v1:127.0.0.1:6805/1656381175] exists,up 3e85ea0b-35f2-48ad-a15a-187736a5768b 2024-08-19T11:41:59.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:41:59.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:41:59.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:41:59.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: test_get_num_pgs: create_rbd_pool 2024-08-19T11:41:59.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:41:59.563 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:41:59.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:41:59.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:42:00.273 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:42:00.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:42:01.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:42:02.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1436: test_get_num_pgs: wait_for_clean 2024-08-19T11:42:02.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:42:02.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:42:02.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:42:02.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:42:02.459 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:42:02.459 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:42:02.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:42:02.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:42:02.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:42:02.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:42:02.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:42:02.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:42:02.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:42:02.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:42:02.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:42:03.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:42:03.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:42:03.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:42:03.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:42:03.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836488 2024-08-19T11:42:03.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836488 2024-08-19T11:42:03.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836488' 2024-08-19T11:42:03.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:42:03.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836488 2024-08-19T11:42:03.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:42:03.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:42:03.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836488 2024-08-19T11:42:03.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:42:03.200 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836488 2024-08-19T11:42:03.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836488 2024-08-19T11:42:03.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-08-19T11:42:03.201 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:42:03.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836486 -lt 21474836488 2024-08-19T11:42:03.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:42:04.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:42:04.619 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:42:05.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836488 -lt 21474836488 2024-08-19T11:42:05.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:42:05.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:42:05.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:42:05.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:42:05.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:42:05.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:42:05.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:42:05.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:42:05.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:42:05.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:42:05.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:42:06.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:42:06.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:42:06.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:42:06.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:42:06.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:42:06.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:42:06.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:42:06.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: get_num_pgs 2024-08-19T11:42:06.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:42:06.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:42:07.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: local num_pgs=4 2024-08-19T11:42:07.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1438: test_get_num_pgs: test 4 -gt 0 2024-08-19T11:42:07.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1439: test_get_num_pgs: teardown td/ceph-helpers 2024-08-19T11:42:07.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:42:07.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:42:07.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:42:07.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:42:07.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:42:07.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:42:07.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:42:07.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:42:07.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:42:07.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:42:07.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:42:07.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:42:07.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:42:07.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:42:07.297 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:42:07.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:42:07.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:42:07.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:42:07.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:42:07.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:42:07.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:42:07.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:42:07.307 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:42:07.307 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:42:07.307 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:42:07.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:42:07.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:42:07.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:42:07.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:42:07.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_get_op_scheduler td/ceph-helpers 2024-08-19T11:42:07.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: test_get_op_scheduler: local dir=td/ceph-helpers 2024-08-19T11:42:07.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: test_get_op_scheduler: setup td/ceph-helpers 2024-08-19T11:42:07.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:42:07.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:42:07.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:42:07.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:42:07.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:42:07.310 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:42:07.310 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:42:07.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:42:07.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:42:07.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:42:07.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:42:07.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:42:07.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:42:07.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:42:07.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:42:07.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:42:07.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:42:07.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:42:07.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:42:07.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:42:07.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:42:07.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:42:07.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:42:07.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:42:07.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:42:07.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:42:07.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:42:07.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:42:07.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:42:07.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:42:07.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:42:07.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:42:07.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:42:07.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:42:07.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:42:07.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:42:07.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:42:07.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:42:07.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:42:07.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: test_get_op_scheduler: run_mon td/ceph-helpers a 2024-08-19T11:42:07.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:42:07.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:42:07.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:42:07.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:42:07.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:42:07.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-08-19T11:42:07.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:42:07.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:42:07.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:42:07.789 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:42:07.789 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:42:07.790 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:42:07.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:42:07.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-19T11:42:08.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:42:08.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:42:08.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:42:08.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:42:08.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:42:08.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:42:08.009 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:42:08.009 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:42:08.009 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:42:08.010 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:42:08.010 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:42:08.010 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:42:08.010 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:42:08.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:42:08.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:42:08.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:42:08.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:42:08.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:42:08.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:42:08.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:42:08.128 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:42:08.128 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:42:08.128 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:42:08.129 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:42:08.129 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:42:08.129 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:42:08.129 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:42:08.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:42:08.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:42:08.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: test_get_op_scheduler: run_mgr td/ceph-helpers x 2024-08-19T11:42:08.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:42:08.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:42:08.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:42:08.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:42:08.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:42:08.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:42:09.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:42:09.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:42:09.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:42:09.145 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:42:09.145 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:42:09.145 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:42:09.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:42:09.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:42:09.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:42:09.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2024-08-19T11:42:09.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:42:09.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:42:09.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:42:09.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:42:09.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:42:09.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:42:09.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:42:09.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:42:09.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:42:09.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:42:09.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:42:09.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:42:09.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:42:09.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:42:09.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:42:09.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:42:09.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:42:09.182 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:42:09.182 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:42:09.182 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=wpq 2024-08-19T11:42:09.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:42:09.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:42:09.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e3b29539-c05e-45c7-ac9b-3f14af530647 2024-08-19T11:42:09.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e3b29539-c05e-45c7-ac9b-3f14af530647' 2024-08-19T11:42:09.185 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 e3b29539-c05e-45c7-ac9b-3f14af530647 2024-08-19T11:42:09.185 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:42:09.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRL8NmqtMJDBAAgDjJy8HDB3BX88lNckkcoA== 2024-08-19T11:42:09.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRL8NmqtMJDBAAgDjJy8HDB3BX88lNckkcoA=="}' 2024-08-19T11:42:09.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e3b29539-c05e-45c7-ac9b-3f14af530647 -i td/ceph-helpers/0/new.json 2024-08-19T11:42:09.511 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:42:09.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:42:09.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQCRL8NmqtMJDBAAgDjJy8HDB3BX88lNckkcoA== --osd-uuid e3b29539-c05e-45c7-ac9b-3f14af530647 2024-08-19T11:42:09.571 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:09.569+0000 7fb4e036e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:42:09.588 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:09.589+0000 7fb4e036e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:42:09.590 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:09.589+0000 7fb4e036e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:42:09.590 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:09.589+0000 7fb4e036e7c0 -1 bdev(0x5578a5f19c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:42:09.590 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:09.589+0000 7fb4e036e7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:42:16.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:42:16.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:42:16.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:42:16.380 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:42:16.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:42:16.966 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:42:16.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:42:16.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2024-08-19T11:42:16.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:42:16.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:42:16.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:42:16.999 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:16.997+0000 7f67dfa5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:42:17.032 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:17.033+0000 7f67dfa5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:42:17.034 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:17.033+0000 7f67dfa5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:42:17.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:42:17.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:42:17.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:42:17.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:42:17.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:42:17.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:17.440 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:42:17.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:42:17.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:17.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:42:17.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:18.279 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:18.277+0000 7f67dfa5e7c0 -1 Falling back to public interface 2024-08-19T11:42:18.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:18.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:18.871 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:42:18.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:42:18.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:18.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:42:19.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:20.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:20.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:20.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:42:20.292 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:42:20.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:42:20.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:20.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:21.187 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:21.185+0000 7f67dfa5e7c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:42:21.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:21.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:21.724 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:42:21.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:42:21.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:21.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:42:22.166 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/569435216,v1:127.0.0.1:6803/569435216] [v2:127.0.0.1:6804/569435216,v1:127.0.0.1:6805/569435216] exists,up e3b29539-c05e-45c7-ac9b-3f14af530647 2024-08-19T11:42:22.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:42:22.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:42:22.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:42:22.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: test_get_op_scheduler: get_op_scheduler 0 2024-08-19T11:42:22.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: get_op_scheduler: local id=0 2024-08-19T11:42:22.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: get_op_scheduler: get_config osd 0 osd_op_queue 2024-08-19T11:42:22.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-08-19T11:42:22.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-08-19T11:42:22.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-08-19T11:42:22.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-08-19T11:42:22.169 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-08-19T11:42:22.169 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-19T11:42:22.169 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-19T11:42:22.169 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:42:22.169 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:42:22.169 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:42:22.169 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-osd.0.asok 2024-08-19T11:42:22.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:42:22.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-osd.0.asok config get osd_op_queue 2024-08-19T11:42:22.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: test_get_op_scheduler: test wpq = wpq 2024-08-19T11:42:22.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2024-08-19T11:42:22.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:42:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:42:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T11:42:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:42:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-08-19T11:42:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:42:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:42:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:42:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:42:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-08-19T11:42:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-08-19T11:42:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:42:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:42:22.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:42:22.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:42:22.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:42:22.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:42:22.295 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:42:22.295 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:42:22.295 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:42:22.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:42:22.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:42:22.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:42:22.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:42:22.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:42:22.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:42:22.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:42:22.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:42:22.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:42:22.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:42:22.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:42:22.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:42:22.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-08-19T11:42:22.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-08-19T11:42:22.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:42:22.297 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 35a87310-4cc3-4bd3-abe8-f4108fb11341 2024-08-19T11:42:22.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=35a87310-4cc3-4bd3-abe8-f4108fb11341 2024-08-19T11:42:22.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 35a87310-4cc3-4bd3-abe8-f4108fb11341' 2024-08-19T11:42:22.297 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:42:22.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCeL8Nma8jLEhAAL70cu74AGvuuo3DnrcxaDA== 2024-08-19T11:42:22.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCeL8Nma8jLEhAAL70cu74AGvuuo3DnrcxaDA=="}' 2024-08-19T11:42:22.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 35a87310-4cc3-4bd3-abe8-f4108fb11341 -i td/ceph-helpers/1/new.json 2024-08-19T11:42:22.901 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:42:22.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-08-19T11:42:22.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCeL8Nma8jLEhAAL70cu74AGvuuo3DnrcxaDA== --osd-uuid 35a87310-4cc3-4bd3-abe8-f4108fb11341 2024-08-19T11:42:22.954 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:22.953+0000 7f6ccff4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:42:22.989 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:22.989+0000 7f6ccff4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:42:22.991 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:22.989+0000 7f6ccff4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:42:22.991 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:22.989+0000 7f6ccff4b7c0 -1 bdev(0x55f5a6a77c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-08-19T11:42:22.991 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:22.989+0000 7f6ccff4b7c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-08-19T11:42:30.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-08-19T11:42:30.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:42:30.255 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-08-19T11:42:30.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T11:42:30.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:42:30.838 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-08-19T11:42:30.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T11:42:30.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-08-19T11:42:30.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:42:30.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:42:30.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:42:30.864 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:30.865+0000 7f216d5787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:42:30.867 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:30.865+0000 7f216d5787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:42:30.868 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:30.869+0000 7f216d5787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:42:31.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T11:42:31.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:42:31.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T11:42:31.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:42:31.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:42:31.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:31.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:42:31.356 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:42:31.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:31.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:31.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:32.135 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:32.133+0000 7f216d5787c0 -1 Falling back to public interface 2024-08-19T11:42:32.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:32.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:32.786 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:42:32.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:42:32.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:32.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:33.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:34.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:34.220 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:42:34.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:34.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:42:34.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:34.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:34.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:35.134 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:42:35.133+0000 7f216d5787c0 -1 osd.1 0 log_to_monitors true 2024-08-19T11:42:35.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:35.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:35.665 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:42:35.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:42:35.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:35.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:36.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:37.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:37.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:37.501 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:42:37.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:42:37.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:37.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:38.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:39.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:39.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:39.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:42:39.147 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:42:39.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:39.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:39.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:40.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:40.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:40.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:42:40.625 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:42:40.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:40.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:41.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:42.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:42.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:42.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:42:42.130 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:42:42.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:42.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:42.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:43.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:43.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:43.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:42:43.695 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:42:43.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:43.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:44.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:45.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:45.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:45.307 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:42:45.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:42:45.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:45.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:45.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:46.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:46.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:46.746 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:42:46.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:42:46.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:46.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:47.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:48.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:48.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:48.181 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:42:48.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:42:48.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:48.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:48.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:49.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:49.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:49.730 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:42:49.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:42:49.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:49.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:50.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:51.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:51.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:51.217 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:42:51.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:42:51.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:51.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:51.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:52.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:52.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:52.906 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:42:52.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:42:52.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:52.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:53.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:54.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:54.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:54.404 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:42:54.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:42:54.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:54.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:56.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:56.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:56.444 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:42:56.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:42:56.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:56.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:57.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:42:58.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:42:58.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:42:58.218 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:42:58.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:42:58.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:42:58.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:42:59.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:00.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:00.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:00.264 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:43:00.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:43:00.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:00.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:43:01.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:02.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:02.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:02.140 INFO:tasks.workunit.client.0.smithi130.stdout:19 2024-08-19T11:43:02.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T11:43:02.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:02.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:43:02.680 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2663193116,v1:127.0.0.1:6811/2663193116] [v2:127.0.0.1:6812/2663193116,v1:127.0.0.1:6813/2663193116] exists,up 35a87310-4cc3-4bd3-abe8-f4108fb11341 2024-08-19T11:43:02.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:43:02.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:43:02.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:43:02.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: test_get_op_scheduler: get_op_scheduler 1 2024-08-19T11:43:02.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: get_op_scheduler: local id=1 2024-08-19T11:43:02.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: get_op_scheduler: get_config osd 1 osd_op_queue 2024-08-19T11:43:02.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-08-19T11:43:02.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=1 2024-08-19T11:43:02.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-08-19T11:43:02.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-08-19T11:43:02.684 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.1 2024-08-19T11:43:02.687 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-19T11:43:02.687 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-19T11:43:02.687 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:43:02.688 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:02.688 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:02.688 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-osd.1.asok 2024-08-19T11:43:02.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:43:02.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-osd.1.asok config get osd_op_queue 2024-08-19T11:43:02.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2024-08-19T11:43:02.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: test_get_op_scheduler: teardown td/ceph-helpers 2024-08-19T11:43:02.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:43:02.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:43:02.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:43:02.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:43:02.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:43:02.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:43:02.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:43:02.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:43:02.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:43:02.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:43:02.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:43:02.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:43:02.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:43:02.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:43:02.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:43:02.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:43:02.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:43:02.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:43:02.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:43:02.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:43:02.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:43:02.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:43:02.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:43:02.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:02.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:02.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:43:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:43:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:43:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:43:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2024-08-19T11:43:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-08-19T11:43:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1461: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2024-08-19T11:43:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:43:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:43:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:43:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:43:02.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:43:02.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:43:02.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:43:02.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:43:02.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:43:02.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:43:02.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:43:02.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:43:02.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:43:02.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:43:02.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:43:02.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:43:02.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:43:02.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:43:02.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:43:02.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:43:02.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:43:02.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:43:02.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:43:02.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:43:02.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:43:02.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:02.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:02.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:43:02.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:43:02.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:43:02.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:43:02.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:43:02.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:02.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:02.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:43:02.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:43:02.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:43:02.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:43:02.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:43:02.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1462: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:43:02.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:43:02.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:43:02.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:43:02.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:43:02.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:43:02.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:43:03.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:43:03.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:43:03.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:43:03.438 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:43:03.438 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:03.438 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:03.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:43:03.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:43:03.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:43:03.659 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:43:03.659 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:43:03.659 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:43:03.659 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:43:03.659 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:43:03.659 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:43:03.659 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:43:03.659 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:43:03.660 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:43:03.660 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:03.660 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:03.660 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:43:03.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:43:03.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:43:03.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:43:03.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:43:03.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:43:03.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:43:03.781 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:43:03.781 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:43:03.781 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:43:03.781 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:43:03.781 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:43:03.781 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:03.781 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:03.781 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:43:03.781 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:43:03.781 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:43:03.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1463: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-08-19T11:43:03.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:43:03.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:43:03.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:43:03.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:43:03.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:43:03.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:43:04.792 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:43:04.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:43:04.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:43:04.793 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:43:04.793 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:04.793 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:04.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:43:04.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:43:04.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:43:04.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1464: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-08-19T11:43:04.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:43:04.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:43:04.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:43:04.836 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:43:04.837 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:43:04.837 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:04.837 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:04.837 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:43:04.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:43:04.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:43:04.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:43:04.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:43:04.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:43:04.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:43:04.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:43:04.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:43:04.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:43:04.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:43:04.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:43:04.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:43:04.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:43:04.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:43:04.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae65e4c8-e011-472a-bd68-a21dab3263b7 2024-08-19T11:43:04.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ae65e4c8-e011-472a-bd68-a21dab3263b7' 2024-08-19T11:43:04.839 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 ae65e4c8-e011-472a-bd68-a21dab3263b7 2024-08-19T11:43:04.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:43:04.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDIL8NmBaoeMxAAcU9qVhhcwq/wXQaa6sf7pA== 2024-08-19T11:43:04.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDIL8NmBaoeMxAAcU9qVhhcwq/wXQaa6sf7pA=="}' 2024-08-19T11:43:04.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae65e4c8-e011-472a-bd68-a21dab3263b7 -i td/ceph-helpers/0/new.json 2024-08-19T11:43:05.151 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:43:05.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:43:05.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIL8NmBaoeMxAAcU9qVhhcwq/wXQaa6sf7pA== --osd-uuid ae65e4c8-e011-472a-bd68-a21dab3263b7 2024-08-19T11:43:05.201 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:05.200+0000 7f4f3f0127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:43:05.219 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:05.216+0000 7f4f3f0127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:43:05.220 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:05.220+0000 7f4f3f0127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:43:05.221 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:05.220+0000 7f4f3f0127c0 -1 bdev(0x562390c6e800 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:43:05.221 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:05.220+0000 7f4f3f0127c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:43:11.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:43:11.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:43:11.826 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:43:11.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:43:11.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:43:12.399 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:43:12.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:43:12.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:43:12.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:43:12.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:43:12.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:43:12.434 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:12.428+0000 7f538b8457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:43:12.464 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:12.464+0000 7f538b8457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:43:12.466 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:12.464+0000 7f538b8457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:43:12.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:43:12.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:43:12.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:43:12.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:43:12.865 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:43:12.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:43:12.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:12.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:43:12.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:12.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:13.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:13.494 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:13.492+0000 7f538b8457c0 -1 Falling back to public interface 2024-08-19T11:43:14.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:14.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:14.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:43:14.293 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:43:14.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:14.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:14.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:15.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:15.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:15.715 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:43:15.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:43:15.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:15.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:16.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:16.437 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:16.436+0000 7f538b8457c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:43:17.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:17.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:17.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:43:17.152 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:43:17.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:17.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:17.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:18.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:18.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:18.728 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:43:18.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:43:18.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:18.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:19.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:20.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:20.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:20.200 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:43:20.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:43:20.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:20.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:20.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:21.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:21.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:21.730 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:43:21.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:43:21.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:21.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:23.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:23.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:23.190 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:43:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:43:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:23.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:24.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:24.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:24.814 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:43:24.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:43:24.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:24.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:25.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:26.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:26.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:26.311 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:43:26.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:43:26.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:26.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:26.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:27.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:27.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:27.775 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:43:27.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:43:27.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:27.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:28.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:29.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:29.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:29.201 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:43:29.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:43:29.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:29.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:29.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:30.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:30.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:30.945 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:43:30.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:43:30.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:30.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:31.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:32.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:32.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:32.441 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:43:32.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:43:32.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:32.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:33.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:34.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:34.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:34.589 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:43:34.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:43:34.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:34.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:35.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:36.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:36.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:36.692 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:43:36.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:43:36.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:36.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:37.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:38.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:38.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:38.553 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:43:38.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:43:38.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:38.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:38.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:39.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:39.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:39.983 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:43:39.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:43:39.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:39.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:40.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:43:41.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:43:41.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:43:41.939 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:43:41.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:43:41.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:43:41.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:43:42.465 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2934885275,v1:127.0.0.1:6803/2934885275] [v2:127.0.0.1:6804/2934885275,v1:127.0.0.1:6805/2934885275] exists,up ae65e4c8-e011-472a-bd68-a21dab3263b7 2024-08-19T11:43:42.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:43:42.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:43:42.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:43:42.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1465: test_get_osd_id_used_by_pgs: create_rbd_pool 2024-08-19T11:43:42.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:43:42.892 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:43:42.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:43:42.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:43:43.459 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:43:43.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:43:44.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:43:46.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1466: test_get_osd_id_used_by_pgs: wait_for_clean 2024-08-19T11:43:46.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:43:46.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:43:46.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:43:46.078 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:43:46.079 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:43:46.079 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:43:46.079 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:43:46.079 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:43:46.079 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:43:46.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:43:46.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:43:46.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:43:46.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:43:46.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:43:46.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:43:46.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:43:46.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:43:46.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:43:46.640 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:43:46.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836488 2024-08-19T11:43:46.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836488 2024-08-19T11:43:46.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836488' 2024-08-19T11:43:46.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:43:46.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:43:46.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836488 2024-08-19T11:43:46.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:43:46.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836488 2024-08-19T11:43:46.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:43:46.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836488 2024-08-19T11:43:46.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-08-19T11:43:46.809 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836488 2024-08-19T11:43:46.810 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:43:47.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836486 -lt 21474836488 2024-08-19T11:43:47.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:43:48.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:43:48.218 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:43:48.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836488 -lt 21474836488 2024-08-19T11:43:48.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:43:48.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:43:48.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:43:49.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:43:49.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:43:49.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:43:49.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:43:49.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:43:49.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:43:49.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:43:49.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:43:49.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:43:49.675 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:43:49.675 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:43:49.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:43:50.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:43:50.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:43:50.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:43:50.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-08-19T11:43:50.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: uniq 2024-08-19T11:43:50.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-08-19T11:43:50.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-08-19T11:43:50.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-08-19T11:43:50.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: local osd_ids=0 2024-08-19T11:43:50.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: test_get_osd_id_used_by_pgs: test 0 = 0 2024-08-19T11:43:50.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-08-19T11:43:50.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:43:50.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:43:50.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:43:50.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:43:50.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:43:50.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:43:50.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:43:50.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:43:50.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:43:50.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:43:50.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:43:50.743 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:43:50.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:43:50.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:43:50.744 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:43:50.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:43:50.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:43:50.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:43:50.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:43:50.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:43:50.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:43:50.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:43:50.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:43:50.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:50.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:50.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:43:50.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:43:50.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:43:50.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:43:50.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_get_osds td/ceph-helpers 2024-08-19T11:43:50.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: local dir=td/ceph-helpers 2024-08-19T11:43:50.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: setup td/ceph-helpers 2024-08-19T11:43:50.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:43:50.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:43:50.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:43:50.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:43:50.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:43:50.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:43:50.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:43:50.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:43:50.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:43:50.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:43:50.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:43:50.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:43:50.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:43:50.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:43:50.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:43:50.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:43:50.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:43:50.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:43:50.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:43:50.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:43:50.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:43:50.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:43:50.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:43:50.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:43:50.762 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:43:50.762 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:50.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:50.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:43:50.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:43:50.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:43:50.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:43:50.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:43:50.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:50.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:50.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:43:50.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:43:50.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:43:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:43:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:43:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-08-19T11:43:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:43:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:43:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:43:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:43:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:43:50.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-08-19T11:43:51.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:43:51.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:43:51.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:43:51.247 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:43:51.248 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:51.248 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:51.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:43:51.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-08-19T11:43:51.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:43:51.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:43:51.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:43:51.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:43:51.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:43:51.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:43:51.459 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:43:51.459 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:43:51.459 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:43:51.460 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:43:51.460 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:51.460 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:51.460 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:43:51.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:43:51.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:43:51.579 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:43:51.579 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:43:51.579 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:43:51.579 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:43:51.579 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:43:51.579 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:43:51.580 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:43:51.580 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:43:51.580 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:43:51.580 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:51.580 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:51.580 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:43:51.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:43:51.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:43:51.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: test_get_osds: run_mgr td/ceph-helpers x 2024-08-19T11:43:51.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:43:51.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:43:51.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:43:51.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:43:51.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:43:51.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:43:52.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:43:52.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:43:52.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:43:52.603 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:43:52.603 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:52.603 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:52.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:43:52.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:43:52.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:43:52.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: test_get_osds: run_osd td/ceph-helpers 0 2024-08-19T11:43:52.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:43:52.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:43:52.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:43:52.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:43:52.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:43:52.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:43:52.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:43:52.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:43:52.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:43:52.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:43:52.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:43:52.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:43:52.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:43:52.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:43:52.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:43:52.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:43:52.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:43:52.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:43:52.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:43:52.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:43:52.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:43:52.640 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:43:52.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4c7eb68b-4311-4c10-9913-80d7b4b3b55e 2024-08-19T11:43:52.641 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 4c7eb68b-4311-4c10-9913-80d7b4b3b55e 2024-08-19T11:43:52.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4c7eb68b-4311-4c10-9913-80d7b4b3b55e' 2024-08-19T11:43:52.641 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:43:52.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD4L8NmA+VOJxAA2Z9W/RvKMePVZhclriIOJA== 2024-08-19T11:43:52.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD4L8NmA+VOJxAA2Z9W/RvKMePVZhclriIOJA=="}' 2024-08-19T11:43:52.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4c7eb68b-4311-4c10-9913-80d7b4b3b55e -i td/ceph-helpers/0/new.json 2024-08-19T11:43:52.960 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:43:52.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:43:52.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD4L8NmA+VOJxAA2Z9W/RvKMePVZhclriIOJA== --osd-uuid 4c7eb68b-4311-4c10-9913-80d7b4b3b55e 2024-08-19T11:43:53.007 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:53.003+0000 7f8bd0c797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:43:53.030 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:53.027+0000 7f8bd0c797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:43:53.032 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:53.031+0000 7f8bd0c797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:43:53.032 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:53.031+0000 7f8bd0c797c0 -1 bdev(0x561e273f7c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:43:53.032 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:43:53.031+0000 7f8bd0c797c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:43:59.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:43:59.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:43:59.465 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:43:59.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:43:59.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:44:00.032 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:44:00.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:44:00.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:44:00.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:44:00.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:44:00.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:44:00.064 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:00.063+0000 7f657b4e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:44:00.090 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:00.087+0000 7f657b4e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:44:00.092 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:00.091+0000 7f657b4e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:44:00.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:44:00.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:44:00.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:44:00.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:44:00.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:44:00.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:00.529 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:44:00.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:44:00.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:00.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:00.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:01.590 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:01.587+0000 7f657b4e97c0 -1 Falling back to public interface 2024-08-19T11:44:01.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:01.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:01.924 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:44:01.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:44:01.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:01.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:02.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:03.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:03.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:03.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:44:03.369 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:44:03.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:03.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:03.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:04.456 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:04.455+0000 7f657b4e97c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:44:04.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:04.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:04.794 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:44:04.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:44:04.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:04.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:05.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:06.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:06.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:06.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:44:06.302 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:44:06.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:06.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:06.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:07.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:07.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:07.797 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:44:07.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:44:07.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:07.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:08.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:09.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:09.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:09.376 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:44:09.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:44:09.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:09.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:09.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:10.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:10.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:10.809 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:44:10.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:44:10.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:10.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:11.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:12.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:12.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:12.331 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:44:12.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:44:12.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:12.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:12.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:13.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:13.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:13.781 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:44:13.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:44:13.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:13.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:14.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:15.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:15.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:15.343 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:44:15.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:44:15.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:15.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:15.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:16.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:16.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:16.808 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:44:16.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:44:16.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:16.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:17.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:18.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:18.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:18.376 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:44:18.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:44:18.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:18.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:19.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:20.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:20.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:20.332 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:44:20.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:44:20.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:20.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:21.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:22.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:22.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:22.531 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:44:22.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:44:22.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:22.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:23.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:24.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:24.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:24.352 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:44:24.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:44:24.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:24.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:25.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:26.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:26.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:26.547 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:44:26.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:44:26.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:26.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:27.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:28.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:28.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:28.398 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:44:28.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:44:28.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:28.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:29.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:30.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:30.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:30.156 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:44:30.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:44:30.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:30.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:44:30.594 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1906317756,v1:127.0.0.1:6803/1906317756] [v2:127.0.0.1:6804/1906317756,v1:127.0.0.1:6805/1906317756] exists,up 4c7eb68b-4311-4c10-9913-80d7b4b3b55e 2024-08-19T11:44:30.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:44:30.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:44:30.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:44:30.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1037: test_get_osds: run_osd td/ceph-helpers 1 2024-08-19T11:44:30.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:44:30.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:44:30.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-19T11:44:30.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:44:30.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-08-19T11:44:30.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:44:30.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:44:30.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:44:30.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:44:30.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-08-19T11:44:30.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-08-19T11:44:30.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:44:30.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:44:30.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:44:30.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:44:30.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:44:30.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:44:30.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:44:30.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:44:30.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:44:30.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-08-19T11:44:30.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:44:30.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0e0374b9-82da-4ea7-9de0-7f04006c8019 2024-08-19T11:44:30.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0e0374b9-82da-4ea7-9de0-7f04006c8019' 2024-08-19T11:44:30.601 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 0e0374b9-82da-4ea7-9de0-7f04006c8019 2024-08-19T11:44:30.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:44:30.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAeMMNm+3InJRAA7sDLZTeazluA2/z0emnsDA== 2024-08-19T11:44:30.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAeMMNm+3InJRAA7sDLZTeazluA2/z0emnsDA=="}' 2024-08-19T11:44:30.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0e0374b9-82da-4ea7-9de0-7f04006c8019 -i td/ceph-helpers/1/new.json 2024-08-19T11:44:31.205 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:44:31.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-08-19T11:44:31.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAeMMNm+3InJRAA7sDLZTeazluA2/z0emnsDA== --osd-uuid 0e0374b9-82da-4ea7-9de0-7f04006c8019 2024-08-19T11:44:31.256 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:31.255+0000 7f1ac76aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:44:31.296 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:31.295+0000 7f1ac76aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:44:31.298 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:31.295+0000 7f1ac76aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:44:31.298 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:31.295+0000 7f1ac76aa7c0 -1 bdev(0x55e4da0b5c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-08-19T11:44:31.298 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:31.295+0000 7f1ac76aa7c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-08-19T11:44:37.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-08-19T11:44:37.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:44:37.892 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-08-19T11:44:37.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-19T11:44:37.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:44:38.477 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-08-19T11:44:38.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-19T11:44:38.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:44:38.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:44:38.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:44:38.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:44:38.510 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:38.506+0000 7f3d67e227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:44:38.539 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:38.534+0000 7f3d67e227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:44:38.540 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:38.538+0000 7f3d67e227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:44:38.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-19T11:44:38.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:44:38.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-19T11:44:38.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:44:38.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:44:38.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:38.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:44:38.912 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:44:38.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:38.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:44:39.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:39.766 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:39.762+0000 7f3d67e227c0 -1 Falling back to public interface 2024-08-19T11:44:40.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:40.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:40.327 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:44:40.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:44:40.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:40.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:44:40.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:41.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:41.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:41.763 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:44:41.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:44:41.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:41.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:44:42.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:42.644 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:44:42.642+0000 7f3d67e227c0 -1 osd.1 0 log_to_monitors true 2024-08-19T11:44:43.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:43.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:43.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:44:43.209 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:44:43.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:43.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:44:43.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:44.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:44.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:44.882 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:44:44.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:44:44.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:44.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:44:45.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:46.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:46.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:46.333 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:44:46.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:44:46.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:46.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:44:46.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:47.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:47.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:47.780 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:44:47.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:44:47.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:47.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:44:48.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:49.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:49.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:49.402 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:44:49.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:44:49.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:49.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:44:50.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:51.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:51.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:51.064 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:44:51.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:44:51.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:51.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:44:51.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:52.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:52.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:52.664 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:44:52.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:44:52.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:52.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:44:53.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:54.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:54.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:54.134 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:44:54.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:44:54.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:54.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:44:54.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:55.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:55.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:55.692 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:44:55.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:44:55.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:55.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:44:56.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:57.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:57.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:57.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:44:57.229 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:44:57.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:57.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:44:58.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:44:59.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:44:59.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:44:59.448 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:44:59.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:44:59.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:44:59.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:45:00.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:01.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:01.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:01.645 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:45:01.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:45:01.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:01.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:45:02.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:03.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:03.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:03.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:45:03.630 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:45:03.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:03.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:45:04.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:05.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:05.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:05.494 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:45:05.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:45:05.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:05.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:45:06.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:07.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:07.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:07.836 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:45:07.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:45:07.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:07.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:45:09.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:10.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:10.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:10.035 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:45:10.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:45:10.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:10.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-19T11:45:10.474 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1005417397,v1:127.0.0.1:6811/1005417397] [v2:127.0.0.1:6812/1005417397,v1:127.0.0.1:6813/1005417397] exists,up 0e0374b9-82da-4ea7-9de0-7f04006c8019 2024-08-19T11:45:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:45:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:45:10.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:45:10.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: test_get_osds: create_rbd_pool 2024-08-19T11:45:10.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:45:10.944 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:45:10.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:45:10.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:45:11.482 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:45:11.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:45:12.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:45:14.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1039: test_get_osds: wait_for_clean 2024-08-19T11:45:14.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:45:14.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:45:14.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:45:14.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:45:14.876 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:45:14.876 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:45:14.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:45:14.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:45:14.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:45:14.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:45:14.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:45:14.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:45:14.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:45:14.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:45:14.998 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:45:15.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-19T11:45:15.465 INFO:tasks.workunit.client.0.smithi130.stderr:1' 2024-08-19T11:45:15.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:45:15.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:45:15.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:45:15.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836499 2024-08-19T11:45:15.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836499 2024-08-19T11:45:15.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836499' 2024-08-19T11:45:15.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:45:15.662 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-19T11:45:15.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672969 2024-08-19T11:45:15.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672969 2024-08-19T11:45:15.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836499 1-42949672969' 2024-08-19T11:45:15.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:45:15.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836499 2024-08-19T11:45:15.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:45:15.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:45:15.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836499 2024-08-19T11:45:15.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:45:15.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836499 2024-08-19T11:45:15.828 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836499 2024-08-19T11:45:15.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-08-19T11:45:15.829 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:45:16.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836497 -lt 21474836499 2024-08-19T11:45:16.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:45:17.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:45:17.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:45:17.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836499 -lt 21474836499 2024-08-19T11:45:17.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:45:17.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672969 2024-08-19T11:45:17.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:45:17.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-19T11:45:17.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672969 2024-08-19T11:45:17.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:45:17.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672969 2024-08-19T11:45:17.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-08-19T11:45:17.740 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672969 2024-08-19T11:45:17.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-19T11:45:18.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672969 -lt 42949672969 2024-08-19T11:45:18.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:45:18.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:45:18.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:45:18.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:45:18.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:45:18.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:45:18.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:45:18.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:45:18.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:45:18.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:45:18.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:45:19.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:45:19.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:45:19.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:45:19.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:45:19.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:45:19.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:45:19.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:45:19.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1040: test_get_osds: create_rbd_pool 2024-08-19T11:45:19.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:45:20.238 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:45:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:45:20.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:45:21.117 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:45:21.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:45:22.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:45:23.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: get_osds rbd GROUP 2024-08-19T11:45:23.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2024-08-19T11:45:23.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=rbd 2024-08-19T11:45:23.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=GROUP 2024-08-19T11:45:23.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map rbd GROUP 2024-08-19T11:45:23.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-19T11:45:24.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-08-19T11:45:24.289 INFO:tasks.workunit.client.0.smithi130.stderr:0' 2024-08-19T11:45:24.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-08-19T11:45:24.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1042: test_get_osds: teardown td/ceph-helpers 2024-08-19T11:45:24.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:45:24.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:45:24.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:45:24.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:45:24.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:45:24.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:45:24.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:45:24.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:45:24.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:45:24.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:45:24.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:45:24.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:45:24.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:45:24.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:45:24.400 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:45:24.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:45:24.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:45:24.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:45:24.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:45:24.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:45:24.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:45:24.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:45:24.415 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:45:24.415 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:45:24.415 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:45:24.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:45:24.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:45:24.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:45:24.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:45:24.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_get_pg td/ceph-helpers 2024-08-19T11:45:24.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: local dir=td/ceph-helpers 2024-08-19T11:45:24.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: test_get_pg: setup td/ceph-helpers 2024-08-19T11:45:24.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:45:24.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:45:24.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:45:24.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:45:24.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:45:24.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:45:24.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:45:24.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:45:24.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:45:24.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:45:24.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:45:24.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:45:24.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:45:24.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:45:24.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:45:24.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:45:24.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:45:24.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:45:24.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:45:24.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:45:24.422 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:45:24.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:45:24.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:45:24.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:45:24.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:45:24.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:45:24.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:45:24.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:45:24.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:45:24.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:45:24.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:45:24.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:45:24.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:45:24.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:45:24.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:45:24.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:45:24.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:45:24.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:45:24.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:45:24.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:45:24.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:45:24.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:45:24.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:45:24.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:45:24.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:45:24.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:45:24.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:45:24.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:45:24.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:45:24.901 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:45:24.901 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:45:24.901 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:45:24.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:45:24.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:45:25.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:45:25.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:45:25.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:45:25.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:45:25.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:45:25.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:45:25.112 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:45:25.112 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:45:25.112 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:45:25.112 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:45:25.112 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:45:25.113 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:45:25.113 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:45:25.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:45:25.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:45:25.223 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:45:25.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:45:25.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:45:25.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:45:25.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:45:25.224 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:45:25.224 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:45:25.225 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:45:25.225 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:45:25.225 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:45:25.225 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:45:25.225 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:45:25.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:45:25.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:45:25.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: test_get_pg: run_mgr td/ceph-helpers x 2024-08-19T11:45:25.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:45:25.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:45:25.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:45:25.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:45:25.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:45:25.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:45:26.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:45:26.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:45:26.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:45:26.250 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:45:26.250 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:45:26.251 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:45:26.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:45:26.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:45:26.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:45:26.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: test_get_pg: run_osd td/ceph-helpers 0 2024-08-19T11:45:26.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:45:26.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:45:26.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:45:26.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:45:26.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:45:26.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:45:26.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:45:26.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:45:26.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:45:26.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:45:26.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:45:26.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:45:26.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:45:26.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:45:26.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:45:26.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:45:26.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:45:26.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:45:26.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:45:26.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:45:26.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:45:26.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:45:26.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43dc2cd4-c4c2-4ec9-a103-3ef5a9ba63cb 2024-08-19T11:45:26.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 43dc2cd4-c4c2-4ec9-a103-3ef5a9ba63cb' 2024-08-19T11:45:26.294 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 43dc2cd4-c4c2-4ec9-a103-3ef5a9ba63cb 2024-08-19T11:45:26.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:45:26.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBWMMNmntyhEhAATxZ61lEVjjpNdYoBFi6JrQ== 2024-08-19T11:45:26.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBWMMNmntyhEhAATxZ61lEVjjpNdYoBFi6JrQ=="}' 2024-08-19T11:45:26.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43dc2cd4-c4c2-4ec9-a103-3ef5a9ba63cb -i td/ceph-helpers/0/new.json 2024-08-19T11:45:26.631 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:45:26.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:45:26.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBWMMNmntyhEhAATxZ61lEVjjpNdYoBFi6JrQ== --osd-uuid 43dc2cd4-c4c2-4ec9-a103-3ef5a9ba63cb 2024-08-19T11:45:26.681 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:45:26.682+0000 7f2d40b257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:45:26.709 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:45:26.710+0000 7f2d40b257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:45:26.710 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:45:26.710+0000 7f2d40b257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:45:26.711 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:45:26.710+0000 7f2d40b257c0 -1 bdev(0x556a09bf5c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:45:26.711 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:45:26.710+0000 7f2d40b257c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:45:33.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:45:33.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:45:33.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:45:33.071 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:45:33.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:45:33.699 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:45:33.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:45:33.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:45:33.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:45:33.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:45:33.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:45:33.730 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:45:33.730+0000 7f05c8ffd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:45:33.772 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:45:33.774+0000 7f05c8ffd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:45:33.773 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:45:33.774+0000 7f05c8ffd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:45:34.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:45:34.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:45:34.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:45:34.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:45:34.159 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:45:34.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:45:34.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:34.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:45:34.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:34.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:34.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:34.898 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:45:34.898+0000 7f05c8ffd7c0 -1 Falling back to public interface 2024-08-19T11:45:35.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:35.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:35.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:45:35.683 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:45:35.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:35.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:36.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:37.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:37.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:37.109 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:45:37.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:45:37.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:37.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:37.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:37.777 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:45:37.778+0000 7f05c8ffd7c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:45:38.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:38.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:38.550 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:45:38.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:45:38.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:38.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:39.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:40.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:40.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:40.115 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:45:40.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:45:40.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:40.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:40.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:41.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:41.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:41.613 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:45:41.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:45:41.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:41.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:42.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:43.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:43.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:43.044 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:45:43.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:45:43.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:43.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:43.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:44.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:44.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:44.529 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:45:44.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:45:44.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:44.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:45.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:46.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:46.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:46.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:45:46.116 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:45:46.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:46.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:46.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:47.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:47.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:47.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:45:47.585 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:45:47.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:47.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:48.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:49.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:49.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:49.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:45:49.123 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:45:49.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:49.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:49.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:50.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:50.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:50.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:45:50.561 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:45:50.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:50.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:51.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:52.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:52.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:52.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:45:52.235 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:45:52.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:52.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:52.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:53.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:53.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:53.754 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:45:53.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:45:53.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:53.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:54.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:55.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:55.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:55.973 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:45:55.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:45:55.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:55.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:56.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:57.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:57.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:45:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:45:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:57.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:45:58.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:45:59.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:45:59.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:45:59.728 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:45:59.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:45:59.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:45:59.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:00.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:01.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:01.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:01.603 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:46:01.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:46:01.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:01.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:02.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:03.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:03.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:03.535 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:46:03.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:46:03.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:03.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:03.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:04.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:04.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:04.985 INFO:tasks.workunit.client.0.smithi130.stdout:19 2024-08-19T11:46:04.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T11:46:04.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:04.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:05.407 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1812010896,v1:127.0.0.1:6803/1812010896] [v2:127.0.0.1:6804/1812010896,v1:127.0.0.1:6805/1812010896] exists,up 43dc2cd4-c4c2-4ec9-a103-3ef5a9ba63cb 2024-08-19T11:46:05.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:46:05.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:46:05.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:46:05.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: test_get_pg: create_rbd_pool 2024-08-19T11:46:05.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:46:05.832 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:46:05.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:46:05.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:46:06.414 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:46:06.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:46:07.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:46:08.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: test_get_pg: wait_for_clean 2024-08-19T11:46:08.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:46:08.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:46:08.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:46:08.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:46:08.717 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:46:08.717 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:46:08.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:46:08.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:46:08.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:46:08.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:46:08.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:46:08.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:46:08.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:46:08.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:46:08.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:46:09.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:46:09.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:46:09.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:46:09.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:46:09.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836489 2024-08-19T11:46:09.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836489 2024-08-19T11:46:09.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836489' 2024-08-19T11:46:09.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:46:09.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:46:09.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836489 2024-08-19T11:46:09.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:46:09.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836489 2024-08-19T11:46:09.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:46:09.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836489 2024-08-19T11:46:09.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-08-19T11:46:09.460 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836489 2024-08-19T11:46:09.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:46:09.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836486 -lt 21474836489 2024-08-19T11:46:09.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:46:10.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:46:10.883 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:46:11.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836489 -lt 21474836489 2024-08-19T11:46:11.323 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:46:11.323 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:46:11.323 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:46:11.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:46:11.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:46:11.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:46:11.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:46:11.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:46:11.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:46:11.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:46:11.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:46:12.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:46:12.268 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:46:12.268 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:46:12.268 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:46:12.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:46:12.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:46:12.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:46:12.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: get_pg rbd GROUP 2024-08-19T11:46:12.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-08-19T11:46:12.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=GROUP 2024-08-19T11:46:12.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2024-08-19T11:46:12.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd GROUP 2024-08-19T11:46:12.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-19T11:46:13.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1109: test_get_pg: teardown td/ceph-helpers 2024-08-19T11:46:13.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:46:13.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:46:13.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:46:13.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:46:13.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:46:13.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:46:13.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:46:13.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:46:13.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:46:13.453 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:46:13.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:46:13.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:46:13.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:46:13.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:46:13.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:46:13.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:46:13.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:46:13.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:46:13.457 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:46:13.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:46:13.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:46:13.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:46:13.466 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:46:13.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:46:13.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:46:13.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:46:13.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:46:13.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:46:13.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:46:13.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_get_primary td/ceph-helpers 2024-08-19T11:46:13.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: local dir=td/ceph-helpers 2024-08-19T11:46:13.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: test_get_primary: setup td/ceph-helpers 2024-08-19T11:46:13.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:46:13.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:46:13.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:46:13.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:46:13.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:46:13.469 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:46:13.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:46:13.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:46:13.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:46:13.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:46:13.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:46:13.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:46:13.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:46:13.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:46:13.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:46:13.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:46:13.473 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:46:13.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:46:13.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:46:13.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:46:13.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:46:13.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:46:13.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:46:13.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:46:13.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:46:13.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:46:13.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:46:13.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:46:13.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:46:13.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:46:13.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:46:13.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:46:13.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:46:13.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:46:13.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:46:13.482 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:46:13.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:46:13.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:46:13.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:46:13.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:46:13.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:46:13.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:46:13.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:46:13.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:46:13.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:46:13.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:46:13.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:46:13.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:46:13.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:46:13.962 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:46:13.962 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:46:13.962 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:46:13.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:46:13.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:46:14.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:46:14.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:46:14.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:46:14.192 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:46:14.192 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:46:14.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:46:14.193 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:46:14.193 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:46:14.193 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:46:14.193 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:46:14.193 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:46:14.193 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:46:14.193 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:46:14.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:46:14.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:46:14.309 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:46:14.310 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:46:14.310 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:46:14.310 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:46:14.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:46:14.311 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:46:14.311 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:46:14.311 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:46:14.311 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:46:14.311 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:46:14.311 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:46:14.311 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:46:14.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:46:14.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:46:14.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: test_get_primary: local osd=0 2024-08-19T11:46:14.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: test_get_primary: run_mgr td/ceph-helpers x 2024-08-19T11:46:14.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:46:14.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:46:14.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:46:14.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:46:14.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:46:14.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:46:15.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:46:15.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:46:15.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:46:15.312 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:46:15.312 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:46:15.312 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:46:15.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:46:15.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:46:15.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:46:15.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: test_get_primary: run_osd td/ceph-helpers 0 2024-08-19T11:46:15.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:46:15.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:46:15.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:46:15.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:46:15.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:46:15.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:46:15.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:46:15.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:46:15.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:46:15.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:46:15.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:46:15.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:46:15.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:46:15.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:46:15.350 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:46:15.350 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:46:15.350 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:46:15.350 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:46:15.350 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:46:15.350 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:46:15.350 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:46:15.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:46:15.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9bb6560b-acf8-48bd-9a65-64603bd42625 2024-08-19T11:46:15.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9bb6560b-acf8-48bd-9a65-64603bd42625' 2024-08-19T11:46:15.352 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 9bb6560b-acf8-48bd-9a65-64603bd42625 2024-08-19T11:46:15.353 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:46:15.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHMMNm7gwRFhAAB7KcaYXI9cWsijHGji045Q== 2024-08-19T11:46:15.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHMMNm7gwRFhAAB7KcaYXI9cWsijHGji045Q=="}' 2024-08-19T11:46:15.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9bb6560b-acf8-48bd-9a65-64603bd42625 -i td/ceph-helpers/0/new.json 2024-08-19T11:46:15.650 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:46:15.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:46:15.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHMMNm7gwRFhAAB7KcaYXI9cWsijHGji045Q== --osd-uuid 9bb6560b-acf8-48bd-9a65-64603bd42625 2024-08-19T11:46:15.701 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:46:15.701+0000 7fb3010ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:46:15.718 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:46:15.717+0000 7fb3010ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:46:15.719 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:46:15.721+0000 7fb3010ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:46:15.720 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:46:15.721+0000 7fb3010ae7c0 -1 bdev(0x561995fdbc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:46:15.720 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:46:15.721+0000 7fb3010ae7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:46:22.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:46:22.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:46:22.208 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:46:22.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:46:22.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:46:22.805 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:46:22.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:46:22.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:46:22.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:46:22.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:46:22.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:46:22.838 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:46:22.837+0000 7f8b954087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:46:22.857 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:46:22.857+0000 7f8b954087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:46:22.858 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:46:22.857+0000 7f8b954087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:46:23.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:46:23.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:46:23.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:46:23.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:46:23.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:46:23.239 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:46:23.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:23.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:46:23.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:23.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:23.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:24.354 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:46:24.353+0000 7f8b954087c0 -1 Falling back to public interface 2024-08-19T11:46:24.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:24.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:24.676 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:46:24.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:46:24.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:24.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:25.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:26.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:26.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:26.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:46:26.099 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:46:26.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:26.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:26.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:27.257 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:46:27.257+0000 7f8b954087c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:46:27.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:27.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:27.526 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:46:27.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:46:27.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:27.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:28.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:29.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:29.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:29.129 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:46:29.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:46:29.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:29.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:29.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:30.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:30.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:30.581 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:46:30.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:46:30.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:30.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:31.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:32.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:32.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:32.117 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:46:32.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:46:32.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:32.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:32.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:33.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:33.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:33.603 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:46:33.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:46:33.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:33.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:34.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:35.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:35.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:35.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:46:35.174 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:46:35.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:35.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:35.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:36.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:36.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:36.627 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:46:36.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:46:36.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:36.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:37.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:38.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:38.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:38.218 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:46:38.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:46:38.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:38.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:38.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:39.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:39.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:39.723 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:46:39.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:46:39.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:39.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:40.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:41.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:41.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:41.393 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:46:41.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:46:41.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:41.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:42.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:43.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:43.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:43.306 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:46:43.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:46:43.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:43.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:44.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:45.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:45.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:45.498 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:46:45.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:46:45.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:45.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:46.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:47.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:47.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:47.322 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:46:47.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:46:47.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:47.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:48.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:49.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:49.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:49.114 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:46:49.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:46:49.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:49.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:50.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:51.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:51.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:51.199 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:46:51.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:46:51.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:51.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:52.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:46:53.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:46:53.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:46:53.253 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:46:53.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:46:53.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:46:53.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:46:53.705 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/890411922,v1:127.0.0.1:6803/890411922] [v2:127.0.0.1:6804/890411922,v1:127.0.0.1:6805/890411922] exists,up 9bb6560b-acf8-48bd-9a65-64603bd42625 2024-08-19T11:46:53.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:46:53.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:46:53.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:46:53.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: test_get_primary: create_rbd_pool 2024-08-19T11:46:53.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:46:54.128 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:46:54.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:46:54.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:46:54.765 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:46:54.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:46:55.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:46:57.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: test_get_primary: wait_for_clean 2024-08-19T11:46:57.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:46:57.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:46:57.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:46:57.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:46:57.147 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:46:57.147 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:46:57.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:46:57.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:46:57.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:46:57.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:46:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:46:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:46:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:46:57.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:46:57.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:46:57.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:46:57.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:46:57.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:46:57.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:46:57.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836487 2024-08-19T11:46:57.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836487 2024-08-19T11:46:57.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836487' 2024-08-19T11:46:57.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:46:57.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-08-19T11:46:57.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:46:57.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:46:57.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836487 2024-08-19T11:46:57.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:46:57.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836487 2024-08-19T11:46:57.930 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836487 2024-08-19T11:46:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-08-19T11:46:57.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:46:58.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836487 -lt 21474836487 2024-08-19T11:46:58.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:46:58.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:46:58.392 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:46:58.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:46:58.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:46:58.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:46:58.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:46:58.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:46:58.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:46:58.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:46:58.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:46:59.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:46:59.373 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:46:59.373 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:46:59.374 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:46:59.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:46:59.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:46:59.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:46:59.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: get_primary rbd GROUP 2024-08-19T11:46:59.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-08-19T11:46:59.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-08-19T11:46:59.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-08-19T11:46:59.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-08-19T11:47:00.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: test 0 = 0 2024-08-19T11:47:00.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: test_get_primary: teardown td/ceph-helpers 2024-08-19T11:47:00.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:47:00.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:47:00.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:47:00.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:47:00.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:47:00.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:47:00.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:47:00.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:47:00.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:47:00.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:47:00.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:47:00.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:47:00.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:47:00.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:47:00.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:47:00.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:47:00.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:47:00.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:47:00.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:47:00.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:47:00.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:47:00.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:47:00.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:47:00.662 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:00.662 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:00.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:47:00.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:47:00.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:47:00.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:47:00.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_get_timeout_delays td/ceph-helpers 2024-08-19T11:47:00.664 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: get_timeout_delays 1 2024-08-19T11:47:00.664 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:00.664 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:00.664 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:00.664 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:00.664 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:00.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: test '1 ' = '1 ' 2024-08-19T11:47:00.675 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: get_timeout_delays 5 2024-08-19T11:47:00.675 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:00.675 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:00.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:00.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:00.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:00.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2024-08-19T11:47:00.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: get_timeout_delays 6 2024-08-19T11:47:00.695 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:00.695 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:00.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:00.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:00.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:00.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2024-08-19T11:47:00.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: get_timeout_delays 7 2024-08-19T11:47:00.713 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:00.713 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:00.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:00.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:00.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:00.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2024-08-19T11:47:00.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: get_timeout_delays 8 2024-08-19T11:47:00.737 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:00.737 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:00.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:00.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:00.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:00.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2024-08-19T11:47:00.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 .1 2024-08-19T11:47:00.764 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:00.764 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:00.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:00.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:00.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:00.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2024-08-19T11:47:00.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 1.5 .1 2024-08-19T11:47:00.792 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:00.792 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:00.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:00.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:00.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:00.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2024-08-19T11:47:00.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 5 .1 2024-08-19T11:47:00.829 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:00.829 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:00.829 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:00.830 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:00.830 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:00.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2024-08-19T11:47:00.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 6 .1 2024-08-19T11:47:00.875 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:00.875 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:00.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:00.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:00.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:00.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2024-08-19T11:47:00.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 6.3 .1 2024-08-19T11:47:00.922 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:00.922 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:00.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:00.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:00.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:00.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2024-08-19T11:47:00.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 20 .1 2024-08-19T11:47:00.972 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:00.973 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:01.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2024-08-19T11:47:01.040 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 300 .1 0 2024-08-19T11:47:01.041 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:01.041 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:01.041 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:01.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:01.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:01.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2024-08-19T11:47:01.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 300 .1 10 2024-08-19T11:47:01.142 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:01.142 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:01.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:01.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:01.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:01.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2024-08-19T11:47:01.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:47:01.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_is_clean td/ceph-helpers 2024-08-19T11:47:01.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: local dir=td/ceph-helpers 2024-08-19T11:47:01.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: test_is_clean: setup td/ceph-helpers 2024-08-19T11:47:01.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:47:01.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:47:01.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:47:01.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:47:01.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:47:01.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:47:01.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:47:01.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:47:01.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:47:01.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:47:01.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:47:01.453 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:47:01.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:47:01.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:47:01.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:47:01.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:47:01.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:47:01.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:47:01.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:47:01.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:47:01.457 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:47:01.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:47:01.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:47:01.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:47:01.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:47:01.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:01.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:01.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:47:01.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:47:01.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:47:01.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:47:01.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:47:01.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:01.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:01.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:47:01.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:47:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:47:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:47:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:47:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:47:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:47:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:47:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:47:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:47:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:47:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:47:01.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:47:01.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:47:01.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:47:01.974 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:47:01.975 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:01.975 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:01.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:47:01.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:47:02.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:47:02.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:47:02.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:47:02.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:47:02.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:47:02.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:47:02.213 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:47:02.213 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:47:02.213 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:47:02.214 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:47:02.214 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:02.214 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:02.214 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:47:02.214 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:47:02.214 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:47:02.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:47:02.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:47:02.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:47:02.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:47:02.332 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:47:02.332 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:47:02.332 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:47:02.332 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:47:02.332 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:47:02.332 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:02.332 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:02.332 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:47:02.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:47:02.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:47:02.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: test_is_clean: run_mgr td/ceph-helpers x 2024-08-19T11:47:02.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:47:02.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:47:02.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:47:02.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:47:02.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:47:02.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:47:03.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:47:03.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:47:03.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:47:03.367 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:47:03.367 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:03.367 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:03.367 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:47:03.367 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:47:03.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:47:03.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: test_is_clean: run_osd td/ceph-helpers 0 2024-08-19T11:47:03.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:47:03.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:47:03.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:47:03.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:47:03.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:47:03.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:47:03.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:47:03.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:47:03.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:47:03.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:47:03.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:47:03.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:47:03.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:47:03.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:47:03.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:47:03.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:47:03.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:47:03.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:47:03.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:47:03.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:47:03.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:47:03.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:47:03.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9ddfe61b-c398-4d7d-b452-bdc1a338b85a 2024-08-19T11:47:03.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9ddfe61b-c398-4d7d-b452-bdc1a338b85a' 2024-08-19T11:47:03.409 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 9ddfe61b-c398-4d7d-b452-bdc1a338b85a 2024-08-19T11:47:03.409 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:47:03.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3MMNmimSEGRAAESRBd/xUYWuvCcfHnuRWKA== 2024-08-19T11:47:03.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3MMNmimSEGRAAESRBd/xUYWuvCcfHnuRWKA=="}' 2024-08-19T11:47:03.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9ddfe61b-c398-4d7d-b452-bdc1a338b85a -i td/ceph-helpers/0/new.json 2024-08-19T11:47:03.738 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:47:03.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:47:03.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3MMNmimSEGRAAESRBd/xUYWuvCcfHnuRWKA== --osd-uuid 9ddfe61b-c398-4d7d-b452-bdc1a338b85a 2024-08-19T11:47:03.788 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:03.788+0000 7f3f9fde37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:47:03.807 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:03.808+0000 7f3f9fde37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:47:03.808 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:03.808+0000 7f3f9fde37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:47:03.809 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:03.808+0000 7f3f9fde37c0 -1 bdev(0x563f04f39c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:47:03.809 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:03.808+0000 7f3f9fde37c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:47:10.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:47:10.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:47:10.291 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:47:10.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:47:10.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:47:10.877 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:47:10.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:47:10.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:47:10.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:47:10.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:47:10.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:47:10.909 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:10.908+0000 7fc24c14d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:47:10.929 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:10.928+0000 7fc24c14d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:47:10.930 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:10.928+0000 7fc24c14d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:47:11.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:47:11.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:47:11.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:47:11.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:47:11.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:47:11.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:11.319 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:47:11.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:47:11.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:11.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:11.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:12.206 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:12.204+0000 7fc24c14d7c0 -1 Falling back to public interface 2024-08-19T11:47:12.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:12.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:12.745 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:47:12.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:47:12.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:12.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:13.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:14.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:14.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:14.179 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:47:14.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:47:14.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:14.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:14.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:15.214 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:15.212+0000 7fc24c14d7c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:47:15.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:15.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:15.627 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:47:15.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:47:15.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:15.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:16.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:17.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:17.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:17.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:47:17.080 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:47:17.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:17.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:17.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:18.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:18.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:18.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:47:18.512 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:47:18.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:18.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:19.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:20.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:20.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:20.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:47:20.097 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:47:20.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:20.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:20.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:21.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:21.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:21.585 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:47:21.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:47:21.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:21.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:22.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:23.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:23.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:23.077 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:47:23.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:47:23.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:23.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:23.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:24.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:24.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:24.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:47:24.521 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:47:24.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:24.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:25.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:26.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:26.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:26.085 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:47:26.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:47:26.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:26.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:26.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:27.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:27.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:27.508 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:47:27.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:47:27.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:27.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:28.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:29.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:29.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:29.133 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:47:29.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:47:29.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:29.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:30.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:31.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:31.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:31.345 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:47:31.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:47:31.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:31.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:32.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:33.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:33.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:33.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:47:33.240 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:47:33.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:33.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:34.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:35.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:35.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:35.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:47:35.157 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:47:35.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:35.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:36.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:37.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:37.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:37.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:47:37.145 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:47:37.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:37.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:37.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:38.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:38.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:38.891 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:47:38.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:47:38.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:38.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:40.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:47:41.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:47:41.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:47:41.110 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:47:41.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:47:41.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:47:41.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:47:41.640 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1835093257,v1:127.0.0.1:6803/1835093257] [v2:127.0.0.1:6804/1835093257,v1:127.0.0.1:6805/1835093257] exists,up 9ddfe61b-c398-4d7d-b452-bdc1a338b85a 2024-08-19T11:47:41.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:47:41.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:47:41.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:47:41.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: test_is_clean: create_rbd_pool 2024-08-19T11:47:41.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:47:42.085 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:47:42.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:47:42.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:47:42.631 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:47:42.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:47:43.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:47:45.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: test_is_clean: wait_for_clean 2024-08-19T11:47:45.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:47:45.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:47:45.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:47:45.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:47:45.003 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:47:45.003 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:47:45.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:47:45.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:47:45.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:47:45.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:47:45.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:47:45.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:47:45.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:47:45.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:47:45.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:47:45.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:47:45.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:47:45.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:47:45.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:47:45.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836488 2024-08-19T11:47:45.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836488 2024-08-19T11:47:45.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836488' 2024-08-19T11:47:45.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:47:45.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836488 2024-08-19T11:47:45.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:47:45.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:47:45.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836488 2024-08-19T11:47:45.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:47:45.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836488 2024-08-19T11:47:45.768 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836488 2024-08-19T11:47:45.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-08-19T11:47:45.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:47:46.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836486 -lt 21474836488 2024-08-19T11:47:46.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:47:47.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:47:47.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:47:47.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836488 -lt 21474836488 2024-08-19T11:47:47.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:47:47.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:47:47.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:47:48.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:47:48.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:47:48.303 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:47:48.303 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:47:48.303 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:47:48.303 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:47:48.303 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:47:48.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:47:48.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:47:48.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:47:48.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:47:48.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:47:49.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:47:49.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:47:49.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:47:49.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1569: test_is_clean: is_clean 2024-08-19T11:47:49.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: get_num_pgs 2024-08-19T11:47:49.305 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:47:49.305 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:47:49.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: num_pgs=4 2024-08-19T11:47:49.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: is_clean: test 4 '!=' 0 2024-08-19T11:47:49.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: get_num_active_clean 2024-08-19T11:47:49.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:47:49.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:47:49.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:47:49.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:47:49.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:47:50.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: test 4 = 4 2024-08-19T11:47:50.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: test_is_clean: teardown td/ceph-helpers 2024-08-19T11:47:50.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:47:50.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:47:50.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:47:50.285 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:47:50.285 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:47:50.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:47:50.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:47:50.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:47:50.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:47:50.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:47:50.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:47:50.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:47:50.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:47:50.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:47:50.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:47:50.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:47:50.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:47:50.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:47:50.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:47:50.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:47:50.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:47:50.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:47:50.405 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:47:50.405 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:50.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:50.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:47:50.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:47:50.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_kill_daemon td/ceph-helpers 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:275: test_kill_daemon: local dir=td/ceph-helpers 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:276: test_kill_daemon: setup td/ceph-helpers 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:47:50.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:47:50.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:47:50.409 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:47:50.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:47:50.410 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:47:50.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:47:50.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:47:50.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:47:50.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:47:50.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:47:50.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:47:50.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:47:50.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:47:50.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:47:50.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:47:50.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:47:50.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:50.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:50.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:47:50.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:47:50.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:47:50.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:47:50.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:47:50.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:50.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:50.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:47:50.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:47:50.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:47:50.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:47:50.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:47:50.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:277: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:47:50.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:47:50.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:47:50.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:47:50.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:47:50.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:47:50.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:47:50.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:47:50.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:47:50.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:47:50.898 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:47:50.898 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:50.898 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:50.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:47:50.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:47:51.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:47:51.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:47:51.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:47:51.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:47:51.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:47:51.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:47:51.144 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:47:51.144 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:47:51.144 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:47:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:47:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:47:51.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:47:51.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:47:51.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:47:51.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:47:51.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:47:51.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:47:51.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:47:51.261 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:47:51.261 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:47:51.261 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:47:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:47:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:47:51.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:47:51.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:47:51.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:278: test_kill_daemon: run_mgr td/ceph-helpers x 2024-08-19T11:47:51.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:47:51.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:47:51.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:47:51.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:47:51.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:47:51.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:47:52.283 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:47:52.283 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:47:52.283 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:47:52.283 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:47:52.283 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:52.283 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:52.283 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:47:52.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:47:52.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:47:52.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:279: test_kill_daemon: run_osd td/ceph-helpers 0 2024-08-19T11:47:52.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:47:52.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:47:52.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:47:52.350 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:47:52.350 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:47:52.350 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:47:52.350 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:47:52.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:47:52.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:47:52.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:47:52.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:47:52.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:47:52.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:47:52.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:47:52.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:47:52.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:47:52.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:47:52.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:47:52.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:47:52.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:47:52.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:47:52.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9b349e7f-a0f3-4bfd-befd-d634884c54f6 2024-08-19T11:47:52.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9b349e7f-a0f3-4bfd-befd-d634884c54f6' 2024-08-19T11:47:52.352 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 9b349e7f-a0f3-4bfd-befd-d634884c54f6 2024-08-19T11:47:52.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:47:52.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDoMMNmDns2FhAAgW3TaKyXzrlizjjAhhjpdg== 2024-08-19T11:47:52.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDoMMNmDns2FhAAgW3TaKyXzrlizjjAhhjpdg=="}' 2024-08-19T11:47:52.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9b349e7f-a0f3-4bfd-befd-d634884c54f6 -i td/ceph-helpers/0/new.json 2024-08-19T11:47:52.669 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:47:52.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:47:52.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDoMMNmDns2FhAAgW3TaKyXzrlizjjAhhjpdg== --osd-uuid 9b349e7f-a0f3-4bfd-befd-d634884c54f6 2024-08-19T11:47:52.724 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:52.723+0000 7f771c6237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:47:52.738 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:52.735+0000 7f771c6237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:47:52.739 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:52.739+0000 7f771c6237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:47:52.739 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:52.739+0000 7f771c6237c0 -1 bdev(0x55e534b79c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:47:52.740 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:52.739+0000 7f771c6237c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:47:59.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:47:59.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:47:59.266 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:47:59.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:47:59.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:47:59.837 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:47:59.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:47:59.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:47:59.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:47:59.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:47:59.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:47:59.869 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:59.867+0000 7f0bc01af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:47:59.871 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:59.871+0000 7f0bc01af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:47:59.872 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:47:59.871+0000 7f0bc01af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:48:00.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:48:00.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:48:00.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:48:00.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:48:00.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:48:00.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:00.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:48:00.294 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:48:00.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:00.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:00.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:00.914 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:48:00.911+0000 7f0bc01af7c0 -1 Falling back to public interface 2024-08-19T11:48:01.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:01.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:01.777 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:48:01.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:48:01.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:01.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:02.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:03.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:03.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:03.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:48:03.230 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:48:03.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:03.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:03.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:03.704 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:48:03.703+0000 7f0bc01af7c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:48:04.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:04.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:04.681 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:48:04.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:48:04.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:04.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:05.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:06.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:06.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:06.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:48:06.201 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:48:06.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:06.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:06.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:07.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:07.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:07.797 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:48:07.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:48:07.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:07.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:08.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:09.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:09.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:09.246 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:48:09.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:48:09.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:09.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:09.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:10.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:10.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:10.800 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:48:10.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:48:10.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:10.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:11.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:12.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:12.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:12.269 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:48:12.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:48:12.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:12.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:12.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:13.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:13.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:13.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:48:13.807 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:48:13.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:13.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:14.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:15.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:15.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:15.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:48:15.283 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:48:15.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:15.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:15.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:16.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:16.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:16.829 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:48:16.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:48:16.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:16.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:17.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:18.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:18.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:18.461 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:48:18.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:48:18.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:18.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:19.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:20.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:20.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:20.292 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:48:20.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:48:20.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:20.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:21.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:22.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:22.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:22.104 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:48:22.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:48:22.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:22.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:23.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:24.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:24.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:24.060 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:48:24.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:48:24.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:24.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:24.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:25.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:25.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:25.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:48:25.929 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:48:25.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:25.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:26.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:27.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:27.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:27.811 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:48:27.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:48:27.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:27.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:28.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:29.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:29.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:29.554 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:48:29.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:48:29.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:29.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:30.059 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2832444070,v1:127.0.0.1:6803/2832444070] [v2:127.0.0.1:6804/2832444070,v1:127.0.0.1:6805/2832444070] exists,up 9b349e7f-a0f3-4bfd-befd-d634884c54f6 2024-08-19T11:48:30.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:48:30.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:48:30.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:48:30.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:281: test_kill_daemon: name_prefix=osd 2024-08-19T11:48:30.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: find td/ceph-helpers 2024-08-19T11:48:30.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: grep 'osd[^/]*\.pid' 2024-08-19T11:48:30.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-08-19T11:48:30.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:289: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2024-08-19T11:48:30.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-08-19T11:48:30.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=118628 2024-08-19T11:48:30.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=0 2024-08-19T11:48:30.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local delays=1 2024-08-19T11:48:30.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-08-19T11:48:30.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-08-19T11:48:30.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 118628 2024-08-19T11:48:30.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-08-19T11:48:30.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-08-19T11:48:30.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 1 2024-08-19T11:48:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-08-19T11:48:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 118628 2024-08-19T11:48:31.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-08-19T11:48:31.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-08-19T11:48:31.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0 2024-08-19T11:48:31.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 1 2024-08-19T11:48:31.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:293: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2024-08-19T11:48:31.069 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-08-19T11:48:31.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=118628 2024-08-19T11:48:31.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-08-19T11:48:31.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-08-19T11:48:31.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-08-19T11:48:31.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-08-19T11:48:31.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 118628 2024-08-19T11:48:31.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-08-19T11:48:31.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-08-19T11:48:31.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-08-19T11:48:31.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-08-19T11:48:31.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 118628 2024-08-19T11:48:31.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-08-19T11:48:31.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-08-19T11:48:31.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2024-08-19T11:48:31.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-08-19T11:48:31.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 118628 2024-08-19T11:48:31.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-08-19T11:48:31.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-08-19T11:48:31.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-08-19T11:48:31.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: name_prefix=mgr 2024-08-19T11:48:31.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: find td/ceph-helpers 2024-08-19T11:48:31.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: grep 'mgr[^/]*\.pid' 2024-08-19T11:48:31.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-08-19T11:48:31.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2024-08-19T11:48:31.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2024-08-19T11:48:31.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=117681 2024-08-19T11:48:31.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-08-19T11:48:31.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-08-19T11:48:31.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-08-19T11:48:31.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-08-19T11:48:31.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 117681 2024-08-19T11:48:31.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-08-19T11:48:31.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-08-19T11:48:31.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-08-19T11:48:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-08-19T11:48:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 117681 2024-08-19T11:48:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-08-19T11:48:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-08-19T11:48:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-08-19T11:48:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: name_prefix=mon 2024-08-19T11:48:31.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: find td/ceph-helpers 2024-08-19T11:48:31.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: grep 'mon[^/]*\.pid' 2024-08-19T11:48:31.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-08-19T11:48:31.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:309: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2024-08-19T11:48:31.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mon.a.pid 2024-08-19T11:48:31.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=117596 2024-08-19T11:48:31.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-08-19T11:48:31.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-08-19T11:48:31.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-08-19T11:48:31.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-08-19T11:48:31.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 117596 2024-08-19T11:48:31.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-08-19T11:48:31.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-08-19T11:48:31.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-08-19T11:48:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-08-19T11:48:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 117596 2024-08-19T11:48:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-08-19T11:48:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-08-19T11:48:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-08-19T11:48:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:310: test_kill_daemon: timeout 5 ceph status 2024-08-19T11:48:36.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:313: test_kill_daemon: teardown td/ceph-helpers 2024-08-19T11:48:36.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:48:36.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:48:36.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:48:36.596 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:48:36.596 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:48:36.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:48:36.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:48:36.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:48:36.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:48:36.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:48:36.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:48:36.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:48:36.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:48:36.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:48:36.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:48:36.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:48:36.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:48:36.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:48:36.608 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:48:36.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:48:36.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:48:36.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:48:36.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:48:36.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:48:36.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:48:36.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:48:36.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:48:36.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:48:36.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:48:36.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_kill_daemons td/ceph-helpers 2024-08-19T11:48:36.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:366: test_kill_daemons: local dir=td/ceph-helpers 2024-08-19T11:48:36.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:367: test_kill_daemons: setup td/ceph-helpers 2024-08-19T11:48:36.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:48:36.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:48:36.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:48:36.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:48:36.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:48:36.620 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:48:36.620 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:48:36.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:48:36.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:48:36.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:48:36.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:48:36.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:48:36.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:48:36.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:48:36.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:48:36.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:48:36.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:48:36.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:48:36.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:48:36.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:48:36.626 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:48:36.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:48:36.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:48:36.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:48:36.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:48:36.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:48:36.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:48:36.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:48:36.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:48:36.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:48:36.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:48:36.632 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:48:36.632 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:48:36.632 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:48:36.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:48:36.634 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:48:36.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:48:36.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:48:36.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:48:36.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:48:36.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:48:36.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:48:36.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:48:36.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:48:36.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:48:36.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:48:37.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:48:37.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:48:37.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:48:37.111 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:48:37.111 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:48:37.111 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:48:37.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:48:37.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:48:37.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:48:37.329 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:48:37.329 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:48:37.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:48:37.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:48:37.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:48:37.331 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:48:37.331 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:48:37.331 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:48:37.331 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:48:37.331 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:48:37.331 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:48:37.331 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:48:37.332 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:48:37.332 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:48:37.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:48:37.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:48:37.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:48:37.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:48:37.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:48:37.451 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:48:37.451 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:48:37.451 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:48:37.451 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:48:37.452 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:48:37.452 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:48:37.452 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:48:37.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:48:37.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:48:37.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:369: test_kill_daemons: run_mgr td/ceph-helpers x 2024-08-19T11:48:37.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:48:37.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:48:37.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:48:37.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:48:37.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:48:37.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:48:38.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:48:38.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:48:38.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:48:38.490 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:48:38.490 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:48:38.490 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:48:38.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:48:38.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:48:38.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:48:38.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:370: test_kill_daemons: run_osd td/ceph-helpers 0 2024-08-19T11:48:38.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:48:38.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:48:38.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:48:38.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:48:38.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:48:38.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:48:38.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:48:38.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:48:38.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:48:38.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:48:38.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:48:38.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:48:38.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:48:38.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:48:38.563 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:48:38.563 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:48:38.563 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:48:38.563 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:48:38.563 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:48:38.563 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:48:38.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:48:38.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:48:38.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e464337e-f212-44a1-aed1-3d3561988086 2024-08-19T11:48:38.565 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 e464337e-f212-44a1-aed1-3d3561988086 2024-08-19T11:48:38.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e464337e-f212-44a1-aed1-3d3561988086' 2024-08-19T11:48:38.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:48:38.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAWMcNmhlLQIhAAsVYexQ1ZG3GW2zrMC328oQ== 2024-08-19T11:48:38.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAWMcNmhlLQIhAAsVYexQ1ZG3GW2zrMC328oQ=="}' 2024-08-19T11:48:38.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e464337e-f212-44a1-aed1-3d3561988086 -i td/ceph-helpers/0/new.json 2024-08-19T11:48:38.875 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:48:38.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:48:38.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAWMcNmhlLQIhAAsVYexQ1ZG3GW2zrMC328oQ== --osd-uuid e464337e-f212-44a1-aed1-3d3561988086 2024-08-19T11:48:38.917 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:48:38.914+0000 7f65aac527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:48:38.943 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:48:38.938+0000 7f65aac527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:48:38.944 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:48:38.942+0000 7f65aac527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:48:38.945 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:48:38.942+0000 7f65aac527c0 -1 bdev(0x55814254dc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:48:38.945 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:48:38.942+0000 7f65aac527c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:48:45.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:48:45.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:48:45.213 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:48:45.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:48:45.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:48:45.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:48:45.795 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:48:45.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:48:45.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:48:45.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:48:45.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:48:45.827 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:48:45.826+0000 7fbf7e59a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:48:45.828 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:48:45.830+0000 7fbf7e59a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:48:45.830 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:48:45.830+0000 7fbf7e59a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:48:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:48:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:48:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:48:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:48:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:48:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:48:46.226 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:48:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:46.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:47.086 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:48:47.086+0000 7fbf7e59a7c0 -1 Falling back to public interface 2024-08-19T11:48:47.705 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:48:47.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:47.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:47.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:48:47.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:47.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:48.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:49.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:49.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:49.174 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:48:49.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:48:49.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:49.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:49.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:50.057 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:48:50.058+0000 7fbf7e59a7c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:48:50.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:50.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:50.621 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:48:50.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:48:50.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:50.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:51.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:52.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:52.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:52.329 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:48:52.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:48:52.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:52.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:52.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:54.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:54.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:54.001 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:48:54.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:48:54.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:54.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:54.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:55.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:55.441 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:48:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:48:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:55.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:56.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:56.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:56.951 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:48:56.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:48:56.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:56.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:57.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:58.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:58.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:58.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:48:58.426 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:48:58.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:58.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:48:58.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:48:59.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:48:59.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:48:59.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:48:59.960 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:48:59.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:48:59.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:00.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:01.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:01.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:01.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:49:01.436 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:49:01.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:01.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:02.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:03.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:03.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:03.061 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:49:03.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:49:03.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:03.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:03.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:04.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:04.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:04.543 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:49:04.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:49:04.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:04.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:05.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:06.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:06.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:06.337 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:49:06.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:49:06.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:06.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:07.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:08.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:08.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:08.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:49:08.274 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:49:08.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:08.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:09.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:10.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:10.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:10.125 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:49:10.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:49:10.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:10.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:11.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:12.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:12.265 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:49:12.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:12.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:49:12.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:12.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:13.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:14.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:14.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:14.183 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:49:14.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:49:14.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:14.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:15.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:16.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:16.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:16.038 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:49:16.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:49:16.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:16.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:16.478 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/741037551,v1:127.0.0.1:6803/741037551] [v2:127.0.0.1:6804/741037551,v1:127.0.0.1:6805/741037551] exists,up e464337e-f212-44a1-aed1-3d3561988086 2024-08-19T11:49:16.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:49:16.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:49:16.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:49:16.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:377: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2024-08-19T11:49:16.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:49:16.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:49:16.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:49:16.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:49:16.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:49:17.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 1 2024-08-19T11:49:17.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2024-08-19T11:49:17.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:49:17.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:49:17.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:49:17.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:49:17.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:49:17.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:49:17.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:385: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2024-08-19T11:49:17.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:49:17.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:49:17.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:49:17.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:49:17.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:49:17.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:49:17.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:389: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2024-08-19T11:49:17.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:49:17.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:49:17.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:49:17.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:49:17.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:49:18.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:49:18.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:390: test_kill_daemons: timeout 5 ceph status 2024-08-19T11:49:23.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:391: test_kill_daemons: teardown td/ceph-helpers 2024-08-19T11:49:23.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:49:23.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:49:23.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:49:23.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:49:23.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:49:23.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:49:23.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:49:23.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:49:23.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:49:23.231 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:49:23.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:49:23.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:49:23.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:49:23.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:49:23.234 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:49:23.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:49:23.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:49:23.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:49:23.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:49:23.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:49:23.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:49:23.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:49:23.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:49:23.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:49:23.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:49:23.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:49:23.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:49:23.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:49:23.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:49:23.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_objectstore_tool td/ceph-helpers 2024-08-19T11:49:23.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: local dir=td/ceph-helpers 2024-08-19T11:49:23.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: setup td/ceph-helpers 2024-08-19T11:49:23.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:49:23.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:49:23.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:49:23.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:49:23.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:49:23.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:49:23.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:49:23.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:49:23.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:49:23.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:49:23.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:49:23.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:49:23.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:49:23.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:49:23.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:49:23.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:49:23.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:49:23.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:49:23.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:49:23.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:49:23.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:49:23.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:49:23.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:49:23.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:49:23.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:49:23.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:49:23.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:49:23.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:49:23.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:49:23.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:49:23.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:49:23.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:49:23.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:49:23.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:49:23.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:49:23.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:49:23.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:49:23.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:49:23.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:49:23.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:49:23.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:49:23.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:49:23.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:49:23.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:49:23.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:49:23.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:49:23.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:49:23.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:49:23.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:49:23.738 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:49:23.738 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:49:23.738 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:49:23.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:49:23.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:49:23.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:49:23.958 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:49:23.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:49:23.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:49:23.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:49:23.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:49:23.960 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:49:23.960 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:49:23.960 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:49:23.960 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:49:23.960 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:49:23.960 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:49:23.960 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:49:23.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:49:23.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:49:24.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:49:24.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:49:24.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:49:24.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:49:24.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:49:24.081 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:49:24.081 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:49:24.081 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:49:24.082 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:49:24.082 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:49:24.082 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:49:24.082 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:49:24.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:49:24.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:49:24.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: local osd=0 2024-08-19T11:49:24.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: test_objectstore_tool: run_mgr td/ceph-helpers x 2024-08-19T11:49:24.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:49:24.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:49:24.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:49:24.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:49:24.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:49:24.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:49:25.097 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:49:25.097 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:49:25.097 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:49:25.098 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:49:25.098 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:49:25.098 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:49:25.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:49:25.099 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:49:25.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:49:25.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: test_objectstore_tool: run_osd td/ceph-helpers 0 2024-08-19T11:49:25.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:49:25.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:49:25.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:49:25.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:49:25.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:49:25.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:49:25.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:49:25.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:49:25.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:49:25.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:49:25.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:49:25.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:49:25.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:49:25.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:49:25.136 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:49:25.136 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:49:25.137 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:49:25.137 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:49:25.137 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:49:25.137 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:49:25.137 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:49:25.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:49:25.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:49:25.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:49:25.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:49:25.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:49:25.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:49:25.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:49:25.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:49:25.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:49:25.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:49:25.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:49:25.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:49:25.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:49:25.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:49:25.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d301e1d6-e183-4968-85dc-bfd757481227 2024-08-19T11:49:25.139 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 d301e1d6-e183-4968-85dc-bfd757481227 2024-08-19T11:49:25.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d301e1d6-e183-4968-85dc-bfd757481227' 2024-08-19T11:49:25.140 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:49:25.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBFMcNm38pYCRAAM9TcQBtlJM25GRf8nRVdew== 2024-08-19T11:49:25.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBFMcNm38pYCRAAM9TcQBtlJM25GRf8nRVdew=="}' 2024-08-19T11:49:25.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d301e1d6-e183-4968-85dc-bfd757481227 -i td/ceph-helpers/0/new.json 2024-08-19T11:49:25.452 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:49:25.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:49:25.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFMcNm38pYCRAAM9TcQBtlJM25GRf8nRVdew== --osd-uuid d301e1d6-e183-4968-85dc-bfd757481227 2024-08-19T11:49:25.497 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:49:25.497+0000 7f2cb59e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:49:25.521 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:49:25.521+0000 7f2cb59e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:49:25.522 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:49:25.521+0000 7f2cb59e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:49:25.522 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:49:25.521+0000 7f2cb59e97c0 -1 bdev(0x5622b82e7c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:49:25.523 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:49:25.521+0000 7f2cb59e97c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:49:32.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:49:32.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:49:32.404 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:49:32.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:49:32.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:49:32.983 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:49:32.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:49:32.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:49:32.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:49:32.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:49:32.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:49:33.015 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:49:33.013+0000 7f579f81b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:49:33.052 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:49:33.053+0000 7f579f81b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:49:33.053 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:49:33.053+0000 7f579f81b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:49:33.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:49:33.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:49:33.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:49:33.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:49:33.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:49:33.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:33.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:49:33.396 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:49:33.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:33.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:33.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:34.550 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:49:34.549+0000 7f579f81b7c0 -1 Falling back to public interface 2024-08-19T11:49:34.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:34.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:34.820 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:49:34.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:49:34.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:34.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:35.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:36.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:36.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:36.294 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:49:36.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:49:36.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:36.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:36.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:37.506 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:49:37.505+0000 7f579f81b7c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:49:37.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:37.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:37.718 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:49:37.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:49:37.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:37.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:38.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:39.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:39.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:39.217 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:49:39.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:49:39.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:39.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:39.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:40.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:40.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:40.676 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:49:40.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:49:40.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:40.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:41.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:42.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:42.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:42.139 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:49:42.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:49:42.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:42.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:42.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:43.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:43.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:43.663 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:49:43.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:49:43.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:43.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:44.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:45.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:45.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:45.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:49:45.135 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:49:45.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:45.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:45.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:46.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:46.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:46.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:49:46.736 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:49:46.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:46.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:47.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:48.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:48.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:48.161 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:49:48.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:49:48.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:48.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:48.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:49.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:49.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:49.674 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:49:49.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:49:49.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:49.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:50.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:51.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:51.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:51.280 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:49:51.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:49:51.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:51.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:52.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:53.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:53.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:53.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:49:53.126 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:49:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:54.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:55.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:55.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:55.014 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:49:55.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:49:55.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:55.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:55.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:56.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:56.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:56.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:49:56.955 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:49:56.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:56.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:49:58.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:49:58.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:49:58.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:49:58.792 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:49:58.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:49:58.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:49:59.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:00.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:00.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:00.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:50:00.625 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:50:00.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:00.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:01.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:02.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:02.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:02.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:50:02.887 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:50:02.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:02.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:03.328 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2126233673,v1:127.0.0.1:6803/2126233673] [v2:127.0.0.1:6804/2126233673,v1:127.0.0.1:6805/2126233673] exists,up d301e1d6-e183-4968-85dc-bfd757481227 2024-08-19T11:50:03.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:50:03.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:50:03.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:50:03.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: test_objectstore_tool: create_rbd_pool 2024-08-19T11:50:03.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:50:03.845 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:50:03.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:50:03.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:50:04.380 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:50:04.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:50:05.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:50:06.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: test_objectstore_tool: wait_for_clean 2024-08-19T11:50:06.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:50:06.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:50:06.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:50:06.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:50:06.824 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:50:06.824 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:50:06.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:50:06.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:50:06.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:50:06.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:50:06.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:50:06.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:50:06.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:50:06.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:50:06.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:50:07.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:50:07.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:50:07.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:50:07.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:50:07.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836487 2024-08-19T11:50:07.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836487 2024-08-19T11:50:07.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836487' 2024-08-19T11:50:07.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:50:07.605 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-08-19T11:50:07.605 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:50:07.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:50:07.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836487 2024-08-19T11:50:07.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:50:07.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836487 2024-08-19T11:50:07.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-08-19T11:50:07.609 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836487 2024-08-19T11:50:07.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:50:08.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836487 -lt 21474836487 2024-08-19T11:50:08.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:50:08.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:50:08.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:50:08.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:50:08.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:50:08.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:50:08.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:50:08.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:50:08.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:50:08.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:50:08.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:50:09.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:50:09.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:50:09.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:50:09.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:50:09.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:50:09.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:50:09.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:50:09.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1316: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2024-08-19T11:50:09.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2024-08-19T11:50:09.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: test_objectstore_tool: diff - /etc/group 2024-08-19T11:50:09.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-08-19T11:50:09.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T11:50:09.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T11:50:09.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T11:50:09.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2024-08-19T11:50:09.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-08-19T11:50:09.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T11:50:09.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T11:50:09.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T11:50:09.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-08-19T11:50:09.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:50:09.697 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:50:09.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:50:09.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:50:09.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:50:10.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:50:10.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2024-08-19T11:50:10.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-08-19T11:50:10.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T11:50:10.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T11:50:10.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T11:50:10.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-08-19T11:50:10.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2024-08-19T11:50:10.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2024-08-19T11:50:10.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-08-19T11:50:10.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-19T11:50:10.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-19T11:50:10.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-19T11:50:10.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:50:10.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:50:10.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:50:10.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:50:10.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:50:10.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:50:10.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:50:10.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-19T11:50:10.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-19T11:50:10.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:50:10.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-19T11:50:10.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:50:10.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:50:10.774 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:50:10.774 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:50:10.775 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:50:10.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:50:10.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:50:10.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:50:10.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:50:10.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:50:10.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:50:10.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:50:10.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:50:10.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:50:10.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-19T11:50:10.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-19T11:50:10.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:50:10.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-19T11:50:10.779 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.0 2024-08-19T11:50:10.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:50:10.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-08-19T11:50:10.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-19T11:50:10.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-19T11:50:10.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-19T11:50:10.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-19T11:50:10.807 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:10.804+0000 7f9474cdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:50:10.809 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:10.808+0000 7f9474cdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:50:10.811 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:10.808+0000 7f9474cdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:50:11.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-19T11:50:11.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:50:11.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:50:11.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:50:11.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:50:11.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:11.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:50:11.183 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-08-19T11:50:11.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:11.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:11.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:12.122 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:12.120+0000 7f9474cdd7c0 -1 Falling back to public interface 2024-08-19T11:50:12.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:12.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:12.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:50:12.623 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-08-19T11:50:12.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:12.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:13.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:14.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:14.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:14.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:50:14.042 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-08-19T11:50:14.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:14.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:14.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:15.455 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:15.452+0000 7f9474cdd7c0 -1 osd.0 13 log_to_monitors true 2024-08-19T11:50:15.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:15.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:15.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:50:15.477 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-08-19T11:50:15.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:15.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:15.954 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 up in weight 1 up_from 16 up_thru 7 down_at 14 last_clean_interval [5,13) [v2:127.0.0.1:6802/2655241560,v1:127.0.0.1:6803/2655241560] [v2:127.0.0.1:6804/2655241560,v1:127.0.0.1:6805/2655241560] exists,up d301e1d6-e183-4968-85dc-bfd757481227 2024-08-19T11:50:15.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:50:15.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:50:15.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:50:15.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-08-19T11:50:15.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:50:15.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:50:15.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:50:15.955 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:50:15.955 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:50:15.955 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:50:15.955 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:50:15.956 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:50:15.956 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:50:16.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:50:16.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:50:16.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:50:16.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:50:16.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:50:16.056 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:50:16.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:50:16.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:50:16.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:50:16.554 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:50:16.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=68719476739 2024-08-19T11:50:16.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 68719476739 2024-08-19T11:50:16.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-68719476739' 2024-08-19T11:50:16.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:50:16.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-68719476739 2024-08-19T11:50:16.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:50:16.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:50:16.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-68719476739 2024-08-19T11:50:16.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:50:16.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=68719476739 2024-08-19T11:50:16.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 68719476739' 2024-08-19T11:50:16.711 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 68719476739 2024-08-19T11:50:16.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:50:17.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 1 -lt 68719476739 2024-08-19T11:50:17.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:50:18.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:50:18.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:50:18.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 68719476739 -lt 68719476739 2024-08-19T11:50:18.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:50:18.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:50:18.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:50:19.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:50:19.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:50:19.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:50:19.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:50:19.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:50:19.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:50:19.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:50:19.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:50:19.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:50:19.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:50:19.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:50:19.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:50:20.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:50:20.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:50:20.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:50:20.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1319: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2024-08-19T11:50:20.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-08-19T11:50:20.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-08-19T11:50:20.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-08-19T11:50:20.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-08-19T11:50:20.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2024-08-19T11:50:20.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-08-19T11:50:20.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-08-19T11:50:20.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-08-19T11:50:20.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-08-19T11:50:20.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-08-19T11:50:20.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:50:20.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:50:20.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:50:20.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:50:20.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:50:20.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:50:20.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2024-08-19T11:50:20.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-08-19T11:50:20.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-19T11:50:20.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-19T11:50:20.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-19T11:50:20.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-08-19T11:50:20.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2024-08-19T11:50:20.855 INFO:tasks.workunit.client.0.smithi130.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2024-08-19T11:50:21.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-08-19T11:50:21.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: return 1 2024-08-19T11:50:21.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: return 1 2024-08-19T11:50:21.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1320: test_objectstore_tool: teardown td/ceph-helpers 2024-08-19T11:50:21.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:50:21.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:50:21.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:50:21.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:50:21.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:50:21.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:50:21.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:50:21.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:50:21.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:50:21.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:50:21.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:50:21.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:50:21.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:50:21.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:50:21.265 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:50:21.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:50:21.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:50:21.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:50:21.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:50:21.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:50:21.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:50:21.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:50:21.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:50:21.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:50:21.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:50:21.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:50:21.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:50:21.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:50:21.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:50:21.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_pg_schedule_scrub td/ceph-helpers 2024-08-19T11:50:21.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1948: test_pg_schedule_scrub: local dir=td/ceph-helpers 2024-08-19T11:50:21.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1950: test_pg_schedule_scrub: setup td/ceph-helpers 2024-08-19T11:50:21.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:50:21.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:50:21.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:50:21.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:50:21.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:50:21.283 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:50:21.283 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:50:21.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:50:21.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:50:21.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:50:21.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:50:21.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:50:21.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:50:21.285 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:50:21.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:50:21.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:50:21.285 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:50:21.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:50:21.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:50:21.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:50:21.286 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:50:21.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:50:21.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:50:21.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:50:21.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:50:21.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:50:21.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:50:21.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:50:21.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:50:21.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:50:21.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:50:21.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:50:21.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:50:21.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:50:21.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:50:21.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:50:21.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:50:21.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:50:21.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:50:21.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: test_pg_schedule_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:50:21.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:50:21.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:50:21.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:50:21.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:50:21.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:50:21.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:50:21.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:50:21.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:50:21.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:50:21.783 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:50:21.783 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:50:21.783 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:50:21.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:50:21.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:50:22.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:50:22.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:50:22.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:50:22.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:50:22.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:50:22.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:50:22.012 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:50:22.013 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:50:22.013 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:50:22.013 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:50:22.013 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:50:22.014 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:50:22.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:50:22.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:50:22.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:50:22.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:50:22.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:50:22.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:50:22.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:50:22.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:50:22.129 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:50:22.129 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:50:22.129 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:50:22.129 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:50:22.129 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:50:22.130 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:50:22.130 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:50:22.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:50:22.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:50:22.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1952: test_pg_schedule_scrub: run_mgr td/ceph-helpers x 2024-08-19T11:50:22.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:50:22.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:50:22.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:50:22.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:50:22.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:50:22.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:50:23.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:50:23.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:50:23.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:50:23.164 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:50:23.164 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:50:23.164 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:50:23.164 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:50:23.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:50:23.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:50:23.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1953: test_pg_schedule_scrub: run_osd td/ceph-helpers 0 2024-08-19T11:50:23.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:50:23.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:50:23.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:50:23.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:50:23.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:50:23.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:50:23.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:50:23.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:50:23.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:50:23.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:50:23.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:50:23.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:50:23.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:50:23.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:50:23.204 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:50:23.204 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:50:23.204 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:50:23.204 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:50:23.204 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:50:23.204 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:50:23.204 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:50:23.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:50:23.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:50:23.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:50:23.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:50:23.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:50:23.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:50:23.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:50:23.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:50:23.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:50:23.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:50:23.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:50:23.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:50:23.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:50:23.206 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:50:23.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=490018f9-f3e2-4f93-9a7a-c070f7e61505 2024-08-19T11:50:23.206 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 490018f9-f3e2-4f93-9a7a-c070f7e61505 2024-08-19T11:50:23.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 490018f9-f3e2-4f93-9a7a-c070f7e61505' 2024-08-19T11:50:23.207 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:50:23.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB/McNm9WNkDRAAwGNKj25SRfyvr91SQwkbpg== 2024-08-19T11:50:23.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB/McNm9WNkDRAAwGNKj25SRfyvr91SQwkbpg=="}' 2024-08-19T11:50:23.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 490018f9-f3e2-4f93-9a7a-c070f7e61505 -i td/ceph-helpers/0/new.json 2024-08-19T11:50:23.536 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:50:23.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:50:23.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/McNm9WNkDRAAwGNKj25SRfyvr91SQwkbpg== --osd-uuid 490018f9-f3e2-4f93-9a7a-c070f7e61505 2024-08-19T11:50:23.581 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:23.580+0000 7f26ef5117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:50:23.614 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:23.612+0000 7f26ef5117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:50:23.616 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:23.616+0000 7f26ef5117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:50:23.617 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:23.616+0000 7f26ef5117c0 -1 bdev(0x5563ff2e0800 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:50:23.617 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:23.616+0000 7f26ef5117c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:50:30.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:50:30.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:50:30.206 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:50:30.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:50:30.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:50:30.781 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:50:30.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:50:30.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:50:30.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:50:30.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:50:30.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:50:30.817 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:30.812+0000 7ff5dff827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:50:30.860 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:30.860+0000 7ff5dff827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:50:30.861 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:30.860+0000 7ff5dff827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:50:31.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:50:31.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:50:31.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:50:31.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:50:31.231 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:50:31.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:50:31.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:31.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:50:31.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:31.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:31.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:32.330 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:32.328+0000 7ff5dff827c0 -1 Falling back to public interface 2024-08-19T11:50:32.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:32.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:32.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:50:32.660 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:50:32.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:32.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:33.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:34.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:34.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:34.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:50:34.093 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:50:34.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:34.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:34.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:35.298 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:50:35.296+0000 7ff5dff827c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:50:35.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:35.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:35.519 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:50:35.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:50:35.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:35.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:36.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:37.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:37.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:37.165 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:50:37.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:50:37.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:37.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:37.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:38.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:38.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:38.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:50:38.646 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:50:38.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:38.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:39.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:40.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:40.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:40.131 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:50:40.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:50:40.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:40.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:40.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:41.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:41.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:41.711 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:50:41.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:50:41.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:41.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:42.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:43.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:43.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:43.374 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:50:43.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:50:43.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:43.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:43.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:44.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:44.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:44.827 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:50:44.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:50:44.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:44.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:45.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:46.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:46.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:46.308 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:50:46.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:50:46.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:46.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:46.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:47.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:47.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:47.757 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:50:47.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:50:47.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:47.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:48.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:49.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:49.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:49.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:50:49.291 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:50:49.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:49.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:50.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:51.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:51.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:51.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:50:51.215 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:50:51.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:51.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:51.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:52.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:52.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:52.982 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:50:52.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:50:52.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:52.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:53.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:54.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:54.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:54.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:50:54.775 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:50:54.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:54.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:55.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:56.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:56.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:56.634 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:50:56.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:50:56.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:56.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:57.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:50:58.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:50:58.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:50:58.656 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:50:58.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:50:58.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:50:58.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:50:59.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:00.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:00.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:00.430 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:51:00.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:51:00.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:00.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:00.881 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1396824687,v1:127.0.0.1:6803/1396824687] [v2:127.0.0.1:6804/1396824687,v1:127.0.0.1:6805/1396824687] exists,up 490018f9-f3e2-4f93-9a7a-c070f7e61505 2024-08-19T11:51:00.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:51:00.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:51:00.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:51:00.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: test_pg_schedule_scrub: create_rbd_pool 2024-08-19T11:51:00.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:51:01.427 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:51:01.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:51:01.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:51:02.092 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:51:02.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:51:03.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:51:04.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1955: test_pg_schedule_scrub: wait_for_clean 2024-08-19T11:51:04.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:51:04.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:51:04.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:51:04.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:51:04.485 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:51:04.485 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:51:04.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:51:04.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:51:04.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:51:04.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:51:04.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:51:04.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:51:04.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:51:04.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:51:04.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:51:05.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:51:05.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:51:05.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:51:05.048 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:51:05.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836483 2024-08-19T11:51:05.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836483 2024-08-19T11:51:05.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836483' 2024-08-19T11:51:05.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:51:05.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836483 2024-08-19T11:51:05.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:51:05.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:51:05.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836483 2024-08-19T11:51:05.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:51:05.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836483 2024-08-19T11:51:05.216 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836483 2024-08-19T11:51:05.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2024-08-19T11:51:05.217 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:51:05.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836481 -lt 21474836483 2024-08-19T11:51:05.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:51:06.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:51:06.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:51:07.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836483 -lt 21474836483 2024-08-19T11:51:07.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:51:07.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:51:07.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:51:07.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:51:07.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:51:07.618 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:51:07.618 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:51:07.619 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:51:07.619 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:51:07.619 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:51:07.619 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:51:08.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:51:08.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:51:08.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:51:08.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:51:08.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:51:08.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:51:08.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:51:08.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1956: test_pg_schedule_scrub: pg_schedule_scrub 1.0 2024-08-19T11:51:08.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_schedule_scrub: local pgid=1.0 2024-08-19T11:51:08.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_schedule_scrub: get_last_scrub_stamp 1.0 2024-08-19T11:51:08.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:51:08.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:51:08.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:51:08.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:51:09.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_schedule_scrub: local last_scrub=2024-08-19T11:51:01.836037+0000 2024-08-19T11:51:09.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_schedule_scrub: ceph pg scrub 1.0 2024-08-19T11:51:09.475 INFO:tasks.workunit.client.0.smithi130.stderr:instructing pg 1.0 on osd.0 to scrub 2024-08-19T11:51:09.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_schedule_scrub: wait_for_scrub 1.0 2024-08-19T11:51:01.836037+0000 2024-08-19T11:51:09.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local pgid=1.0 2024-08-19T11:51:09.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local last_scrub=2024-08-19T11:51:01.836037+0000 2024-08-19T11:51:09.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T11:51:09.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i=0 )) 2024-08-19T11:51:09.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 300 )) 2024-08-19T11:51:09.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:51:09.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:51:09.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:51:09.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:51:09.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:51:09.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:51:01.836037+0000 '>' 2024-08-19T11:51:01.836037+0000 2024-08-19T11:51:09.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: wait_for_scrub: sleep 1 2024-08-19T11:51:10.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i++ )) 2024-08-19T11:51:10.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 300 )) 2024-08-19T11:51:10.890 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:51:10.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:51:10.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:51:10.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:51:10.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:51:11.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:51:01.836037+0000 '>' 2024-08-19T11:51:01.836037+0000 2024-08-19T11:51:11.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: wait_for_scrub: sleep 1 2024-08-19T11:51:12.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i++ )) 2024-08-19T11:51:12.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 300 )) 2024-08-19T11:51:12.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:51:12.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:51:12.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:51:12.327 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:51:12.327 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:51:12.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:51:10.408366+0000 '>' 2024-08-19T11:51:01.836037+0000 2024-08-19T11:51:12.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: wait_for_scrub: return 0 2024-08-19T11:51:12.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1957: test_pg_schedule_scrub: kill_daemons td/ceph-helpers KILL osd 2024-08-19T11:51:12.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:51:12.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:51:12.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:51:12.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:51:12.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:51:12.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:51:12.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1958: test_pg_schedule_scrub: TIMEOUT=1 2024-08-19T11:51:12.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1958: test_pg_schedule_scrub: pg_scrub 1.0 2024-08-19T11:51:12.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: local pgid=1.0 2024-08-19T11:51:12.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: get_last_scrub_stamp 1.0 2024-08-19T11:51:12.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:51:12.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:51:12.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:51:12.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:51:13.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: local last_scrub=2024-08-19T11:51:10.408366+0000 2024-08-19T11:51:13.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: pg_scrub: ceph pg scrub 1.0 2024-08-19T11:51:13.689 INFO:tasks.workunit.client.0.smithi130.stderr:instructing pg 1.0 on osd.0 to scrub 2024-08-19T11:51:13.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: pg_scrub: wait_for_scrub 1.0 2024-08-19T11:51:10.408366+0000 2024-08-19T11:51:13.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local pgid=1.0 2024-08-19T11:51:13.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local last_scrub=2024-08-19T11:51:10.408366+0000 2024-08-19T11:51:13.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T11:51:13.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i=0 )) 2024-08-19T11:51:13.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 1 )) 2024-08-19T11:51:13.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:51:13.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:51:13.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:51:13.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:51:13.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:51:14.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:51:10.408366+0000 '>' 2024-08-19T11:51:10.408366+0000 2024-08-19T11:51:14.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: wait_for_scrub: sleep 1 2024-08-19T11:51:15.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i++ )) 2024-08-19T11:51:15.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 1 )) 2024-08-19T11:51:15.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2036: wait_for_scrub: return 1 2024-08-19T11:51:15.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1959: test_pg_schedule_scrub: teardown td/ceph-helpers 2024-08-19T11:51:15.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:51:15.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:51:15.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:51:15.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:51:15.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:51:15.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:51:15.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:51:15.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:51:15.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:51:15.222 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:51:15.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:51:15.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:51:15.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:51:15.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:51:15.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:51:15.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:51:15.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:51:15.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:51:15.227 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:51:15.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:51:15.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:51:15.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:51:15.235 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:51:15.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:51:15.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:51:15.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:51:15.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:51:15.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:51:15.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:51:15.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_pg_scrub td/ceph-helpers 2024-08-19T11:51:15.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: test_pg_scrub: local dir=td/ceph-helpers 2024-08-19T11:51:15.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: test_pg_scrub: setup td/ceph-helpers 2024-08-19T11:51:15.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:51:15.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:51:15.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:51:15.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:51:15.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:51:15.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:51:15.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:51:15.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:51:15.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:51:15.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:51:15.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:51:15.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:51:15.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:51:15.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:51:15.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:51:15.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:51:15.243 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:51:15.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:51:15.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:51:15.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:51:15.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:51:15.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:51:15.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:51:15.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:51:15.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:51:15.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:51:15.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:51:15.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:51:15.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:51:15.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:51:15.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:51:15.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:51:15.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:51:15.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:51:15.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:51:15.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:51:15.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:51:15.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:51:15.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:51:15.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1909: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:51:15.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:51:15.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:51:15.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:51:15.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:51:15.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:51:15.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:51:15.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:51:15.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:51:15.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:51:15.752 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:51:15.752 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:51:15.753 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:51:15.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:51:15.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:51:15.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:51:15.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:51:15.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:51:15.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:51:15.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:51:15.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:51:15.982 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:51:15.983 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:51:15.983 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:51:15.983 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:51:15.983 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:51:15.983 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:51:15.983 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:51:15.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:51:15.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:51:16.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:51:16.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:51:16.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:51:16.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:51:16.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:51:16.101 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:51:16.102 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:51:16.102 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:51:16.102 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:51:16.102 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:51:16.102 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:51:16.102 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:51:16.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:51:16.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:51:16.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: test_pg_scrub: run_mgr td/ceph-helpers x 2024-08-19T11:51:16.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:51:16.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:51:16.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:51:16.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:51:16.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:51:16.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:51:17.133 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:51:17.133 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:51:17.133 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:51:17.133 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:51:17.133 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:51:17.134 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:51:17.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:51:17.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:51:17.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:51:17.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: test_pg_scrub: run_osd td/ceph-helpers 0 2024-08-19T11:51:17.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:51:17.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:51:17.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:51:17.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:51:17.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:51:17.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:51:17.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:51:17.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:51:17.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:51:17.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:51:17.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:51:17.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:51:17.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:51:17.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:51:17.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:51:17.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:51:17.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:51:17.173 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:51:17.173 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:51:17.173 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:51:17.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:51:17.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:51:17.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:51:17.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:51:17.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:51:17.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:51:17.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:51:17.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:51:17.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:51:17.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:51:17.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:51:17.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:51:17.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:51:17.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:51:17.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:51:17.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=86ff9065-5168-4475-adeb-b0a877eff39c 2024-08-19T11:51:17.175 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 86ff9065-5168-4475-adeb-b0a877eff39c 2024-08-19T11:51:17.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 86ff9065-5168-4475-adeb-b0a877eff39c' 2024-08-19T11:51:17.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:51:17.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1McNmj7ufCxAA32Ued1lzgA0um9xlsucViA== 2024-08-19T11:51:17.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1McNmj7ufCxAA32Ued1lzgA0um9xlsucViA=="}' 2024-08-19T11:51:17.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 86ff9065-5168-4475-adeb-b0a877eff39c -i td/ceph-helpers/0/new.json 2024-08-19T11:51:17.525 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:51:17.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:51:17.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1McNmj7ufCxAA32Ued1lzgA0um9xlsucViA== --osd-uuid 86ff9065-5168-4475-adeb-b0a877eff39c 2024-08-19T11:51:17.572 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:51:17.571+0000 7f177411c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:51:17.602 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:51:17.599+0000 7f177411c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:51:17.604 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:51:17.599+0000 7f177411c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:51:17.604 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:51:17.603+0000 7f177411c7c0 -1 bdev(0x5632999c9c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:51:17.604 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:51:17.603+0000 7f177411c7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:51:24.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:51:24.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:51:24.026 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:51:24.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:51:24.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:51:24.596 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:51:24.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:51:24.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:51:24.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:51:24.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:51:24.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:51:24.630 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:51:24.627+0000 7f1515c547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:51:24.665 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:51:24.663+0000 7f1515c547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:51:24.666 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:51:24.663+0000 7f1515c547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:51:25.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:51:25.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:51:25.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:51:25.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:51:25.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:51:25.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:25.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:51:25.081 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:51:25.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:25.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:25.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:25.955 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:51:25.951+0000 7f1515c547c0 -1 Falling back to public interface 2024-08-19T11:51:26.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:26.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:26.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:51:26.499 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:51:26.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:26.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:26.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:27.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:27.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:27.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:51:27.931 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:51:27.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:27.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:28.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:28.974 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:51:28.971+0000 7f1515c547c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:51:29.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:29.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:29.367 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:51:29.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:51:29.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:29.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:29.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:30.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:30.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:30.828 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:51:30.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:51:30.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:30.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:31.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:32.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:32.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:32.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:51:32.276 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:51:32.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:32.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:32.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:33.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:33.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:33.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:51:33.853 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:51:33.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:33.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:34.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:35.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:35.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:35.320 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:51:35.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:51:35.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:35.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:35.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:36.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:36.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:36.843 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:51:36.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:51:36.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:36.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:37.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:38.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:38.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:38.371 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:51:38.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:51:38.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:38.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:38.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:39.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:39.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:39.874 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:51:39.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:51:39.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:39.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:40.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:41.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:41.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:41.320 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:51:41.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:51:41.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:41.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:41.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:42.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:42.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:42.904 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:51:42.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:51:42.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:42.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:43.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:44.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:44.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:44.814 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:51:44.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:51:44.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:44.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:45.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:46.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:46.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:46.792 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:51:46.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:51:46.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:46.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:47.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:48.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:48.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:48.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:51:48.597 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:51:48.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:48.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:49.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:50.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:50.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:50.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:51:50.209 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:51:50.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:50.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:51.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:52.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:52.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:52.068 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:51:52.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:51:52.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:52.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:52.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:53.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:53.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:53.664 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:51:53.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:51:53.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:53.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:54.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:51:55.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:51:55.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:51:55.536 INFO:tasks.workunit.client.0.smithi130.stdout:19 2024-08-19T11:51:55.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-19T11:51:55.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:51:55.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:51:55.955 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1409108216,v1:127.0.0.1:6803/1409108216] [v2:127.0.0.1:6804/1409108216,v1:127.0.0.1:6805/1409108216] exists,up 86ff9065-5168-4475-adeb-b0a877eff39c 2024-08-19T11:51:55.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:51:55.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:51:55.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:51:55.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: test_pg_scrub: create_rbd_pool 2024-08-19T11:51:55.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:51:56.428 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:51:56.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:51:56.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:51:57.006 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:51:57.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:51:58.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:51:59.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1913: test_pg_scrub: wait_for_clean 2024-08-19T11:51:59.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:51:59.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:51:59.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:51:59.327 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:51:59.328 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:51:59.328 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:51:59.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:51:59.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:51:59.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:51:59.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:51:59.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:51:59.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:51:59.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:51:59.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:51:59.446 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:51:59.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:51:59.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:51:59.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:51:59.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:52:00.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836485 2024-08-19T11:52:00.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836485 2024-08-19T11:52:00.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836485' 2024-08-19T11:52:00.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:52:00.057 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836485 2024-08-19T11:52:00.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:52:00.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:52:00.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836485 2024-08-19T11:52:00.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:52:00.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836485 2024-08-19T11:52:00.060 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836485 2024-08-19T11:52:00.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-08-19T11:52:00.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:52:00.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836483 -lt 21474836485 2024-08-19T11:52:00.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:52:01.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:52:01.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:52:01.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836483 -lt 21474836485 2024-08-19T11:52:01.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:52:02.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 299 -eq 0 ']' 2024-08-19T11:52:02.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:52:03.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836486 -lt 21474836485 2024-08-19T11:52:03.415 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:52:03.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:52:03.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:52:04.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:52:04.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:52:04.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:52:04.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:52:04.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:52:04.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:52:04.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:52:04.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:52:04.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:52:04.442 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:52:04.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:52:04.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:52:04.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:52:04.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:52:04.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:52:04.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1914: test_pg_scrub: pg_scrub 1.0 2024-08-19T11:52:04.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: local pgid=1.0 2024-08-19T11:52:04.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: get_last_scrub_stamp 1.0 2024-08-19T11:52:04.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:52:04.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:52:04.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:52:04.989 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:52:05.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: local last_scrub=2024-08-19T11:51:56.828166+0000 2024-08-19T11:52:05.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: pg_scrub: ceph pg scrub 1.0 2024-08-19T11:52:05.832 INFO:tasks.workunit.client.0.smithi130.stderr:instructing pg 1.0 on osd.0 to scrub 2024-08-19T11:52:05.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: pg_scrub: wait_for_scrub 1.0 2024-08-19T11:51:56.828166+0000 2024-08-19T11:52:05.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local pgid=1.0 2024-08-19T11:52:05.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local last_scrub=2024-08-19T11:51:56.828166+0000 2024-08-19T11:52:05.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T11:52:05.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i=0 )) 2024-08-19T11:52:05.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 300 )) 2024-08-19T11:52:05.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:52:05.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:52:05.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:52:05.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:52:05.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:52:06.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:51:56.828166+0000 '>' 2024-08-19T11:51:56.828166+0000 2024-08-19T11:52:06.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: wait_for_scrub: sleep 1 2024-08-19T11:52:07.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i++ )) 2024-08-19T11:52:07.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 300 )) 2024-08-19T11:52:07.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:52:07.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:52:07.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:52:07.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:52:07.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:52:07.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:51:56.828166+0000 '>' 2024-08-19T11:51:56.828166+0000 2024-08-19T11:52:07.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: wait_for_scrub: sleep 1 2024-08-19T11:52:08.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i++ )) 2024-08-19T11:52:08.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 300 )) 2024-08-19T11:52:08.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:52:08.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:52:08.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:52:08.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:52:08.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:52:09.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:51:56.828166+0000 '>' 2024-08-19T11:51:56.828166+0000 2024-08-19T11:52:09.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: wait_for_scrub: sleep 1 2024-08-19T11:52:10.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i++ )) 2024-08-19T11:52:10.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 300 )) 2024-08-19T11:52:10.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:52:10.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:52:10.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:52:10.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:52:10.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:52:10.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:51:56.828166+0000 '>' 2024-08-19T11:51:56.828166+0000 2024-08-19T11:52:10.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: wait_for_scrub: sleep 1 2024-08-19T11:52:11.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i++ )) 2024-08-19T11:52:11.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 300 )) 2024-08-19T11:52:11.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:52:11.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:52:11.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:52:11.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:52:11.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:52:11.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:52:06.403932+0000 '>' 2024-08-19T11:51:56.828166+0000 2024-08-19T11:52:11.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: wait_for_scrub: return 0 2024-08-19T11:52:11.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1915: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2024-08-19T11:52:11.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:52:11.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:52:11.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:52:11.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:52:11.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:52:12.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:52:12.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1916: test_pg_scrub: TIMEOUT=1 2024-08-19T11:52:12.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1916: test_pg_scrub: pg_scrub 1.0 2024-08-19T11:52:12.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: local pgid=1.0 2024-08-19T11:52:12.070 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: get_last_scrub_stamp 1.0 2024-08-19T11:52:12.070 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:52:12.070 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:52:12.070 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:52:12.070 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:52:12.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: local last_scrub=2024-08-19T11:52:06.403932+0000 2024-08-19T11:52:12.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: pg_scrub: ceph pg scrub 1.0 2024-08-19T11:52:12.879 INFO:tasks.workunit.client.0.smithi130.stderr:instructing pg 1.0 on osd.0 to scrub 2024-08-19T11:52:12.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: pg_scrub: wait_for_scrub 1.0 2024-08-19T11:52:06.403932+0000 2024-08-19T11:52:12.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local pgid=1.0 2024-08-19T11:52:12.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local last_scrub=2024-08-19T11:52:06.403932+0000 2024-08-19T11:52:12.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T11:52:12.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i=0 )) 2024-08-19T11:52:12.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 1 )) 2024-08-19T11:52:12.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:52:12.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:52:12.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:52:12.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:52:12.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:52:13.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:52:06.403932+0000 '>' 2024-08-19T11:52:06.403932+0000 2024-08-19T11:52:13.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: wait_for_scrub: sleep 1 2024-08-19T11:52:14.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i++ )) 2024-08-19T11:52:14.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 1 )) 2024-08-19T11:52:14.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2036: wait_for_scrub: return 1 2024-08-19T11:52:14.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: test_pg_scrub: teardown td/ceph-helpers 2024-08-19T11:52:14.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:52:14.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:52:14.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:52:14.308 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:52:14.308 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:52:14.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:52:14.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:52:14.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:52:14.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:52:14.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:52:14.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:52:14.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:52:14.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:52:14.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:52:14.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:52:14.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:52:14.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:52:14.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:52:14.422 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:52:14.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:52:14.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:52:14.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:52:14.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:52:14.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:52:14.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:52:14.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:52:14.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:52:14.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:52:14.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: for func in $funcs 2024-08-19T11:52:14.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: run_tests: test_repair td/ceph-helpers 2024-08-19T11:52:14.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1863: test_repair: local dir=td/ceph-helpers 2024-08-19T11:52:14.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1865: test_repair: setup td/ceph-helpers 2024-08-19T11:52:14.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-08-19T11:52:14.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-08-19T11:52:14.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:52:14.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-19T11:52:14.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:52:14.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:52:14.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:52:14.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:52:14.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:52:14.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:52:14.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:52:14.439 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:52:14.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:52:14.439 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:52:14.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:52:14.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:52:14.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:52:14.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:52:14.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:52:14.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:52:14.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:52:14.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:52:14.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-19T11:52:14.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:52:14.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:52:14.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:52:14.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:52:14.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:52:14.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:52:14.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:52:14.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-08-19T11:52:14.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-19T11:52:14.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:52:14.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:52:14.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71449 2024-08-19T11:52:14.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-19T11:52:14.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-19T11:52:14.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-19T11:52:14.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-08-19T11:52:14.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1866: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:52:14.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-08-19T11:52:14.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-19T11:52:14.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-19T11:52:14.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-19T11:52:14.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-08-19T11:52:14.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:52:14.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-19T11:52:14.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:52:14.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:52:14.936 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:52:14.936 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:52:14.936 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:52:14.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:52:14.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-08-19T11:52:15.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-19T11:52:15.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-19T11:52:15.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:52:15.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:52:15.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-19T11:52:15.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-19T11:52:15.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:52:15.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:52:15.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:52:15.175 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:52:15.175 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:52:15.175 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:52:15.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:52:15.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:52:15.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get fsid 2024-08-19T11:52:15.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-19T11:52:15.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-19T11:52:15.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-19T11:52:15.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-19T11:52:15.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-19T11:52:15.295 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-19T11:52:15.295 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-19T11:52:15.295 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-19T11:52:15.295 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-19T11:52:15.295 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:52:15.296 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:52:15.296 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71449/ceph-mon.a.asok 2024-08-19T11:52:15.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-19T11:52:15.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71449/ceph-mon.a.asok config get mon_host 2024-08-19T11:52:15.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1867: test_repair: run_mgr td/ceph-helpers x 2024-08-19T11:52:15.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-08-19T11:52:15.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-19T11:52:15.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-19T11:52:15.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-19T11:52:15.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-08-19T11:52:15.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-19T11:52:16.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-19T11:52:16.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:52:16.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:52:16.340 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:52:16.340 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:52:16.340 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:52:16.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:52:16.341 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:52:16.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-19T11:52:16.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1868: test_repair: run_osd td/ceph-helpers 0 2024-08-19T11:52:16.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-08-19T11:52:16.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-19T11:52:16.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-19T11:52:16.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-19T11:52:16.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-08-19T11:52:16.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-08-19T11:52:16.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-19T11:52:16.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-19T11:52:16.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-19T11:52:16.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-08-19T11:52:16.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-08-19T11:52:16.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-19T11:52:16.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-19T11:52:16.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-08-19T11:52:16.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-19T11:52:16.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-19T11:52:16.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-19T11:52:16.375 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-19T11:52:16.375 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:52:16.375 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:52:16.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:52:16.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' 2024-08-19T11:52:16.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-19T11:52:16.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-19T11:52:16.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-19T11:52:16.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-08-19T11:52:16.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-08-19T11:52:16.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-19T11:52:16.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-19T11:52:16.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-19T11:52:16.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-19T11:52:16.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-19T11:52:16.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-19T11:52:16.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-08-19T11:52:16.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-19T11:52:16.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3a1da1fd-c3ae-4975-8f80-823152870090 2024-08-19T11:52:16.377 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 3a1da1fd-c3ae-4975-8f80-823152870090 2024-08-19T11:52:16.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3a1da1fd-c3ae-4975-8f80-823152870090' 2024-08-19T11:52:16.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-19T11:52:16.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDwMcNmQGqfFxAAxbInaOyMFCNjRxWbMWRVCg== 2024-08-19T11:52:16.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDwMcNmQGqfFxAAxbInaOyMFCNjRxWbMWRVCg=="}' 2024-08-19T11:52:16.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3a1da1fd-c3ae-4975-8f80-823152870090 -i td/ceph-helpers/0/new.json 2024-08-19T11:52:16.716 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:52:16.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-08-19T11:52:16.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDwMcNmQGqfFxAAxbInaOyMFCNjRxWbMWRVCg== --osd-uuid 3a1da1fd-c3ae-4975-8f80-823152870090 2024-08-19T11:52:16.762 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:52:16.762+0000 7f1b6d2747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:52:16.793 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:52:16.794+0000 7f1b6d2747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:52:16.794 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:52:16.794+0000 7f1b6d2747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:52:16.795 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:52:16.794+0000 7f1b6d2747c0 -1 bdev(0x55909449fc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-08-19T11:52:16.795 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:52:16.794+0000 7f1b6d2747c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-08-19T11:52:23.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-08-19T11:52:23.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-19T11:52:23.287 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-08-19T11:52:23.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-19T11:52:23.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-19T11:52:23.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-19T11:52:23.872 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-08-19T11:52:23.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-19T11:52:23.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-19T11:52:23.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-19T11:52:23.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=afaeed96-27fa-458d-806a-da069322b128 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.71449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-19T11:52:23.905 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:52:23.906+0000 7f64850277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:52:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:52:23.938+0000 7f64850277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:52:23.939 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:52:23.942+0000 7f64850277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-19T11:52:24.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-19T11:52:24.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-19T11:52:24.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-19T11:52:24.319 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-08-19T11:52:24.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-19T11:52:24.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-19T11:52:24.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:24.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-19T11:52:24.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:24.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:24.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:25.198 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:52:25.198+0000 7f64850277c0 -1 Falling back to public interface 2024-08-19T11:52:25.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:25.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:25.803 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-08-19T11:52:25.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-19T11:52:25.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:25.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:26.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:27.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:27.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:27.221 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-08-19T11:52:27.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-19T11:52:27.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:27.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:27.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:28.195 INFO:tasks.workunit.client.0.smithi130.stderr:2024-08-19T11:52:28.194+0000 7f64850277c0 -1 osd.0 0 log_to_monitors true 2024-08-19T11:52:28.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:28.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:28.669 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-08-19T11:52:28.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-19T11:52:28.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:28.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:29.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:30.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:30.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:30.378 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-08-19T11:52:30.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-19T11:52:30.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:30.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:30.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:31.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:31.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:31.845 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-08-19T11:52:31.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-19T11:52:31.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:31.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:32.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:33.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:33.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:33.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-19T11:52:33.385 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-08-19T11:52:33.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:33.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:33.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:34.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:34.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:34.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-19T11:52:34.849 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-08-19T11:52:34.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:34.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:35.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:36.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:36.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:36.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-19T11:52:36.378 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-08-19T11:52:36.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:36.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:36.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:37.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:37.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:37.860 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-08-19T11:52:37.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-19T11:52:37.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:37.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:38.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:39.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:39.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:39.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-19T11:52:39.412 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-08-19T11:52:39.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:39.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:39.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:40.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:40.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:40.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-19T11:52:40.833 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-08-19T11:52:40.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:40.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:41.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:42.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:42.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:42.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-19T11:52:42.497 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-08-19T11:52:42.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:42.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:43.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:44.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:44.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:44.164 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-08-19T11:52:44.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-19T11:52:44.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:44.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:45.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:46.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:46.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:46.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-19T11:52:46.007 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-08-19T11:52:46.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:46.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:46.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:47.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:47.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:47.884 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-08-19T11:52:47.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-19T11:52:47.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:47.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:49.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:50.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:50.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:50.027 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-08-19T11:52:50.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-19T11:52:50.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:50.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:50.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:51.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:51.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:51.856 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-08-19T11:52:51.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-19T11:52:51.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:51.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:52.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-19T11:52:53.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-19T11:52:53.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-19T11:52:53.734 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-08-19T11:52:53.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-19T11:52:53.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-19T11:52:53.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-19T11:52:54.215 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/585919293,v1:127.0.0.1:6803/585919293] [v2:127.0.0.1:6804/585919293,v1:127.0.0.1:6805/585919293] exists,up 3a1da1fd-c3ae-4975-8f80-823152870090 2024-08-19T11:52:54.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-19T11:52:54.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-19T11:52:54.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-19T11:52:54.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1869: test_repair: create_rbd_pool 2024-08-19T11:52:54.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-08-19T11:52:54.667 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-08-19T11:52:54.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-08-19T11:52:54.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-08-19T11:52:55.289 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-08-19T11:52:55.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-19T11:52:56.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-08-19T11:52:57.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1870: test_repair: wait_for_clean 2024-08-19T11:52:57.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-19T11:52:57.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-19T11:52:57.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-19T11:52:57.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-19T11:52:57.545 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-19T11:52:57.546 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-19T11:52:57.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-19T11:52:57.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-19T11:52:57.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-19T11:52:57.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-19T11:52:57.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-19T11:52:57.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-19T11:52:57.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-19T11:52:57.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-19T11:52:57.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-19T11:52:58.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-19T11:52:58.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-19T11:52:58.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-19T11:52:58.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-19T11:52:58.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836488 2024-08-19T11:52:58.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836488 2024-08-19T11:52:58.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836488' 2024-08-19T11:52:58.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-19T11:52:58.335 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836488 2024-08-19T11:52:58.335 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-19T11:52:58.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-19T11:52:58.336 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836488 2024-08-19T11:52:58.336 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-19T11:52:58.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836488 2024-08-19T11:52:58.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-08-19T11:52:58.337 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836488 2024-08-19T11:52:58.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:52:58.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836486 -lt 21474836488 2024-08-19T11:52:58.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-19T11:52:59.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-19T11:52:59.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-19T11:53:00.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836488 -lt 21474836488 2024-08-19T11:53:00.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-19T11:53:00.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:53:00.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:53:00.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-08-19T11:53:00.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-19T11:53:00.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-19T11:53:00.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-19T11:53:00.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-19T11:53:00.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-19T11:53:00.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-19T11:53:00.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-19T11:53:01.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-08-19T11:53:01.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-19T11:53:01.233 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-19T11:53:01.233 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-19T11:53:01.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-08-19T11:53:01.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-19T11:53:01.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-19T11:53:01.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1871: test_repair: repair 1.0 2024-08-19T11:53:01.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-08-19T11:53:01.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-08-19T11:53:01.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:53:01.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:53:01.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:53:01.772 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:53:02.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-08-19T11:52:55.066659+0000 2024-08-19T11:53:02.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-08-19T11:53:02.576 INFO:tasks.workunit.client.0.smithi130.stderr:instructing pg 1.0 on osd.0 to repair 2024-08-19T11:53:02.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-08-19T11:52:55.066659+0000 2024-08-19T11:53:02.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local pgid=1.0 2024-08-19T11:53:02.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local last_scrub=2024-08-19T11:52:55.066659+0000 2024-08-19T11:53:02.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T11:53:02.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i=0 )) 2024-08-19T11:53:02.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 300 )) 2024-08-19T11:53:02.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:53:02.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:53:02.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:53:02.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:53:02.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:53:02.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:52:55.066659+0000 '>' 2024-08-19T11:52:55.066659+0000 2024-08-19T11:53:02.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: wait_for_scrub: sleep 1 2024-08-19T11:53:03.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i++ )) 2024-08-19T11:53:03.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 300 )) 2024-08-19T11:53:03.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:53:03.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:53:03.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:53:03.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:53:03.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:53:04.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:52:55.066659+0000 '>' 2024-08-19T11:52:55.066659+0000 2024-08-19T11:53:04.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: wait_for_scrub: sleep 1 2024-08-19T11:53:05.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i++ )) 2024-08-19T11:53:05.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 300 )) 2024-08-19T11:53:05.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:53:05.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:53:05.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:53:05.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:53:05.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:53:05.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:53:03.403530+0000 '>' 2024-08-19T11:52:55.066659+0000 2024-08-19T11:53:05.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: wait_for_scrub: return 0 2024-08-19T11:53:05.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1859: repair: sleep 2 2024-08-19T11:53:07.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: test_repair: kill_daemons td/ceph-helpers KILL osd 2024-08-19T11:53:07.858 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:53:07.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:53:07.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:53:07.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:53:07.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:53:07.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:53:07.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: test_repair: TIMEOUT=1 2024-08-19T11:53:07.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: test_repair: repair 1.0 2024-08-19T11:53:07.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-08-19T11:53:07.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-08-19T11:53:07.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:53:07.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:53:07.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:53:07.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:53:08.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-08-19T11:53:03.403530+0000 2024-08-19T11:53:08.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-08-19T11:53:08.822 INFO:tasks.workunit.client.0.smithi130.stderr:instructing pg 1.0 on osd.0 to repair 2024-08-19T11:53:08.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-08-19T11:53:03.403530+0000 2024-08-19T11:53:08.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local pgid=1.0 2024-08-19T11:53:08.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local last_scrub=2024-08-19T11:53:03.403530+0000 2024-08-19T11:53:08.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: wait_for_scrub: local sname=last_scrub_stamp 2024-08-19T11:53:08.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i=0 )) 2024-08-19T11:53:08.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 1 )) 2024-08-19T11:53:08.892 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-08-19T11:53:08.892 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-08-19T11:53:08.892 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-08-19T11:53:08.892 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-08-19T11:53:08.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-08-19T11:53:09.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: test 2024-08-19T11:53:03.403530+0000 '>' 2024-08-19T11:53:03.403530+0000 2024-08-19T11:53:09.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: wait_for_scrub: sleep 1 2024-08-19T11:53:10.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i++ )) 2024-08-19T11:53:10.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: (( i < 1 )) 2024-08-19T11:53:10.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2036: wait_for_scrub: return 1 2024-08-19T11:53:10.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1859: repair: sleep 2 2024-08-19T11:53:12.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: test_repair: return 1 2024-08-19T11:53:12.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: run_tests: teardown td/ceph-helpers 1 2024-08-19T11:53:12.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-08-19T11:53:12.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-08-19T11:53:12.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-08-19T11:53:12.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-19T11:53:12.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-19T11:53:12.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-19T11:53:12.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-19T11:53:12.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-19T11:53:12.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-19T11:53:12.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-19T11:53:12.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-19T11:53:12.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-19T11:53:12.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-19T11:53:12.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-19T11:53:12.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-19T11:53:12.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:53:12.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-19T11:53:12.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-19T11:53:12.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T11:53:12.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:53:12.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-08-19T11:53:12.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-08-19T11:53:12.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-08-19T11:53:12.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/ceph-helpers/mgr.x.log td/ceph-helpers/mon.a.log td/ceph-helpers/osd.0.log /home/ubuntu/cephtest/archive/log 2024-08-19T11:53:12.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-08-19T11:53:12.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-19T11:53:12.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-19T11:53:12.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71449 2024-08-19T11:53:12.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71449 2024-08-19T11:53:12.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-19T11:53:12.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-19T11:53:12.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: run_tests: return 1 2024-08-19T11:53:12.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: : exit 1 2024-08-19T11:53:12.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T11:53:12.366 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2024-08-19T11:53:12.367 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-19T11:53:12.670 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on smithi130 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2024-08-19T11:53:12.855 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=40aa2097d6be48c98c509a2bda099c3f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on smithi130 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2024-08-19T11:53:12.857 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-19T11:53:12.867 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-19T11:53:12.867 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-19T11:53:12.927 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-19T11:53:12.938 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-19T11:53:12.938 DEBUG:teuthology.orchestra.run.smithi130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T11:53:12.951 INFO:teuthology.orchestra.run.smithi130.stdout: remote refid st t when poll reach delay offset jitter 2024-08-19T11:53:12.951 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================== 2024-08-19T11:53:12.951 INFO:teuthology.orchestra.run.smithi130.stdout:+hv01.front.sepi 67.205.162.81 3 u 15 64 377 0.067 -3.342 0.287 2024-08-19T11:53:12.951 INFO:teuthology.orchestra.run.smithi130.stdout:+hv02.front.sepi 63.231.80.2 3 u 8 64 377 0.107 -1.987 0.329 2024-08-19T11:53:12.951 INFO:teuthology.orchestra.run.smithi130.stdout:*hv03.front.sepi 74.6.168.72 3 u 16 64 377 0.089 -3.218 0.522 2024-08-19T11:53:12.952 INFO:teuthology.orchestra.run.smithi130.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-08-19T11:53:12.952 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-19T11:53:12.964 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-19T11:53:12.964 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-19T11:53:12.978 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-19T11:53:12.993 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-19T11:53:13.007 INFO:teuthology.task.internal:Duration was 3023.711460 seconds 2024-08-19T11:53:13.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-19T11:53:13.021 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-19T11:53:13.022 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-19T11:53:13.064 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-19T11:53:13.064 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2024-08-19T11:53:13.064 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-19T11:53:13.122 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-19T11:53:13.122 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-19T11:53:13.204 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-19T11:53:13.205 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-19T11:53:13.306 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-19T11:53:13.315 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-19T11:53:13.315 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-19T11:53:13.329 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-19T11:53:13.340 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:53:13.381 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2024-08-19T11:53:13.401 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-19T11:53:13.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T11:53:13.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-19T11:53:13.461 INFO:teuthology.task.internal:Transferring archived files... 2024-08-19T11:53:13.463 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859430/remote/smithi130 2024-08-19T11:53:13.464 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-19T11:53:13.790 INFO:teuthology.task.internal:Removing archive directory... 2024-08-19T11:53:13.791 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-19T11:53:13.830 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-19T11:53:13.847 INFO:teuthology.task.internal:Not uploading archives. 2024-08-19T11:53:13.847 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-19T11:53:13.860 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-19T11:53:13.860 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-19T11:53:13.878 INFO:teuthology.orchestra.run.smithi130.stdout: 658636 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 19 11:53 /home/ubuntu/cephtest 2024-08-19T11:53:13.878 INFO:teuthology.orchestra.run.smithi130.stdout: 788847 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 19 11:12 /home/ubuntu/cephtest/mnt.0 2024-08-19T11:53:13.878 INFO:teuthology.orchestra.run.smithi130.stdout: 790652 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 19 11:33 /home/ubuntu/cephtest/mnt.0/client.0 2024-08-19T11:53:13.879 INFO:teuthology.orchestra.run.smithi130.stdout: 793490 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 19 11:33 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-19T11:53:13.879 INFO:teuthology.orchestra.run.smithi130.stdout: 808392 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Aug 19 11:53 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-08-19T11:53:13.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T11:53:13.880 INFO:teuthology.orchestra.run.smithi130.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-19T11:53:13.880 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi130 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-19T11:53:13.886 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-19T11:53:13.897 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-19T11:53:13.909 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi130 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-19T11:53:13.913 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} duration: 3023.711459875107 failure_reason: 'Command failed (workunit test misc/test-ceph-helpers.sh) on smithi130 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=40aa2097d6be48c98c509a2bda099c3f status: fail success: false 2024-08-19T11:53:13.913 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-19T11:53:14.010 INFO:teuthology.run:FAIL