2024-02-22T02:17:27.062 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-22T02:17:27.062 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} --archive /home/teuthworker/archive/pdhange-2024-02-22_01:13:10-rados-wip-pdhange-testing1-distro-default-smithi/7570447 --owner scheduled_pdhange@teuthology --name pdhange-2024-02-22_01:13:10-rados-wip-pdhange-testing1-distro-default-smithi --verbose -- /home/teuthworker/archive/pdhange-2024-02-22_01:13:10-rados-wip-pdhange-testing1-distro-default-smithi/7570447/orig.config.yaml 2024-02-22T02:17:27.084 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-22T02:17:27.159 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdhange-2024-02-22_01:13:10-rados-wip-pdhange-testing1-distro-default-smithi/7570447 branch: wip-pdhange-testing1 description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} email: null first_in_suite: false job_id: '7570447' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: pdhange-2024-02-22_01:13:10-rados-wip-pdhange-testing1-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-pdhange-testing1 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: dbe7fcec16eddee6239e4dd68c4d0203c7df6461 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: dbe7fcec16eddee6239e4dd68c4d0203c7df6461 workunit: branch: wip-pdhange-testing1 sha1: dbe7fcec16eddee6239e4dd68c4d0203c7df6461 owner: scheduled_pdhange@teuthology priority: 55 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 3672 sha1: dbe7fcec16eddee6239e4dd68c4d0203c7df6461 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-pdhange-testing1 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_dbe7fcec16eddee6239e4dd68c4d0203c7df6461/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: dbe7fcec16eddee6239e4dd68c4d0203c7df6461 targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPheQoPgX42Hb+CI14rS5w8sLfs7Knm3a9aQ03qEo1ZQxQqySdR4ER39y7uTAbr58u9LK7gREjfm9xOj+WnCLVA= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-22_01:13:10 tube: smithi user: pdhange verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-22T02:17:27.159 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_dbe7fcec16eddee6239e4dd68c4d0203c7df6461/qa; will attempt to use it 2024-02-22T02:17:27.160 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_dbe7fcec16eddee6239e4dd68c4d0203c7df6461/qa/tasks 2024-02-22T02:17:27.160 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-22T02:17:27.162 INFO:teuthology.task.internal:Checking packages... 2024-02-22T02:17:27.183 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'dbe7fcec16eddee6239e4dd68c4d0203c7df6461' 2024-02-22T02:17:27.184 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-22T02:17:27.184 INFO:teuthology.packaging:ref: None 2024-02-22T02:17:27.184 INFO:teuthology.packaging:tag: None 2024-02-22T02:17:27.184 INFO:teuthology.packaging:branch: wip-pdhange-testing1 2024-02-22T02:17:27.184 INFO:teuthology.packaging:sha1: dbe7fcec16eddee6239e4dd68c4d0203c7df6461 2024-02-22T02:17:27.184 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-pdhange-testing1 2024-02-22T02:17:27.333 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1633-gdbe7fcec-1jammy 2024-02-22T02:17:27.334 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-22T02:17:27.340 INFO:teuthology.task.internal:no buildpackages task found 2024-02-22T02:17:27.340 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-22T02:17:27.346 INFO:teuthology.task.internal:Saving configuration 2024-02-22T02:17:27.355 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-22T02:17:27.360 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-22T02:17:27.386 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdhange-2024-02-22_01:13:10-rados-wip-pdhange-testing1-distro-default-smithi/7570447', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-22 02:11:32.209018', 'locked_by': 'scheduled_pdhange@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAtZj9Op3zyaTlkchopkY2rGFcD62KxJhR1+H9hMs2IG'} 2024-02-22T02:17:27.386 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-22T02:17:27.393 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-02-22T02:17:27.394 INFO:teuthology.run_tasks:Running task console_log... 2024-02-22T02:17:27.437 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb9b26380d0>, signals=[15]) 2024-02-22T02:17:27.438 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-22T02:17:27.444 INFO:teuthology.task.internal:Opening connections... 2024-02-22T02:17:27.444 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-02-22T02:17:27.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-22T02:17:27.532 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-22T02:17:27.537 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-02-22T02:17:27.656 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-02-22T02:17:27.657 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-02-22T02:17:27.703 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-02-22T02:17:27.703 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="Ubuntu" 2024-02-22T02:17:27.703 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="22.04" 2024-02-22T02:17:27.703 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-02-22T02:17:27.703 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_CODENAME=jammy 2024-02-22T02:17:27.703 INFO:teuthology.orchestra.run.smithi046.stdout:ID=ubuntu 2024-02-22T02:17:27.703 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE=debian 2024-02-22T02:17:27.703 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://www.ubuntu.com/" 2024-02-22T02:17:27.703 INFO:teuthology.orchestra.run.smithi046.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-02-22T02:17:27.703 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-02-22T02:17:27.703 INFO:teuthology.orchestra.run.smithi046.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-02-22T02:17:27.703 INFO:teuthology.orchestra.run.smithi046.stdout:UBUNTU_CODENAME=jammy 2024-02-22T02:17:27.704 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-02-22T02:17:27.725 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-22T02:17:27.733 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-22T02:17:27.739 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-22T02:17:27.739 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-02-22T02:17:27.750 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-22T02:17:27.756 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-22T02:17:27.756 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-02-22T02:17:27.801 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-22T02:17:27.833 INFO:teuthology.run_tasks:Running task kernel... 2024-02-22T02:17:27.846 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-22T02:17:27.846 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-22T02:17:27.846 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-22T02:17:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-22T02:17:27.851 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-02-22T02:17:27.899 INFO:teuthology.orchestra.run.smithi046.stdout:5.15.0-56-generic 2024-02-22T02:17:27.899 INFO:teuthology.task.kernel:Running kernel on smithi046: 5.15.0-56-generic 2024-02-22T02:17:27.899 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get clean 2024-02-22T02:17:28.020 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get update 2024-02-22T02:17:28.170 INFO:teuthology.orchestra.run.smithi046.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-02-22T02:17:28.268 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-22T02:17:28.350 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-02-22T02:17:28.409 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [404 kB] 2024-02-22T02:17:28.487 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,161 kB] 2024-02-22T02:17:28.530 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [213 kB] 2024-02-22T02:17:28.533 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-02-22T02:17:28.533 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.2 kB] 2024-02-22T02:17:28.534 INFO:teuthology.orchestra.run.smithi046.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,401 kB] 2024-02-22T02:17:28.554 INFO:teuthology.orchestra.run.smithi046.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [231 kB] 2024-02-22T02:17:28.557 INFO:teuthology.orchestra.run.smithi046.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-02-22T02:17:28.557 INFO:teuthology.orchestra.run.smithi046.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [592 kB] 2024-02-22T02:17:28.563 INFO:teuthology.orchestra.run.smithi046.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [842 kB] 2024-02-22T02:17:28.575 INFO:teuthology.orchestra.run.smithi046.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [161 kB] 2024-02-22T02:17:28.577 INFO:teuthology.orchestra.run.smithi046.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-02-22T02:17:28.581 INFO:teuthology.orchestra.run.smithi046.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-02-22T02:17:28.581 INFO:teuthology.orchestra.run.smithi046.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-02-22T02:17:28.582 INFO:teuthology.orchestra.run.smithi046.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-02-22T02:17:28.582 INFO:teuthology.orchestra.run.smithi046.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-02-22T02:17:28.714 INFO:teuthology.orchestra.run.smithi046.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-02-22T02:17:28.851 INFO:teuthology.orchestra.run.smithi046.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,377 kB] 2024-02-22T02:17:29.143 INFO:teuthology.orchestra.run.smithi046.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [570 kB] 2024-02-22T02:17:29.182 INFO:teuthology.orchestra.run.smithi046.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [273 kB] 2024-02-22T02:17:29.198 INFO:teuthology.orchestra.run.smithi046.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-02-22T02:17:29.199 INFO:teuthology.orchestra.run.smithi046.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [34.8 kB] 2024-02-22T02:17:29.200 INFO:teuthology.orchestra.run.smithi046.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,431 kB] 2024-02-22T02:17:29.258 INFO:teuthology.orchestra.run.smithi046.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [235 kB] 2024-02-22T02:17:29.264 INFO:teuthology.orchestra.run.smithi046.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-02-22T02:17:29.429 INFO:teuthology.orchestra.run.smithi046.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,049 kB] 2024-02-22T02:17:30.058 INFO:teuthology.orchestra.run.smithi046.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [689 kB] 2024-02-22T02:17:30.095 INFO:teuthology.orchestra.run.smithi046.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [237 kB] 2024-02-22T02:17:30.108 INFO:teuthology.orchestra.run.smithi046.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-02-22T02:17:30.112 INFO:teuthology.orchestra.run.smithi046.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-02-22T02:17:30.114 INFO:teuthology.orchestra.run.smithi046.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-02-22T02:17:30.114 INFO:teuthology.orchestra.run.smithi046.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-02-22T02:17:30.115 INFO:teuthology.orchestra.run.smithi046.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-02-22T02:17:30.115 INFO:teuthology.orchestra.run.smithi046.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-02-22T02:17:30.116 INFO:teuthology.orchestra.run.smithi046.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-02-22T02:17:30.141 INFO:teuthology.orchestra.run.smithi046.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-02-22T02:17:30.305 INFO:teuthology.orchestra.run.smithi046.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-02-22T02:17:30.387 INFO:teuthology.orchestra.run.smithi046.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-02-22T02:17:30.468 INFO:teuthology.orchestra.run.smithi046.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2024-02-22T02:17:30.523 INFO:teuthology.orchestra.run.smithi046.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-02-22T02:17:30.575 INFO:teuthology.orchestra.run.smithi046.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-02-22T02:17:35.105 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 11.6 MB in 2s (4,655 kB/s) 2024-02-22T02:17:36.428 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-02-22T02:17:36.445 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-22T02:17:36.508 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-02-22T02:17:36.700 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2024-02-22T02:17:36.701 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2024-02-22T02:17:36.873 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages were automatically installed and are no longer required: 2024-02-22T02:17:36.873 INFO:teuthology.orchestra.run.smithi046.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-22T02:17:36.873 INFO:teuthology.orchestra.run.smithi046.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-22T02:17:36.874 INFO:teuthology.orchestra.run.smithi046.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-22T02:17:36.874 INFO:teuthology.orchestra.run.smithi046.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-02-22T02:17:36.874 INFO:teuthology.orchestra.run.smithi046.stdout: python2.7-minimal 2024-02-22T02:17:36.874 INFO:teuthology.orchestra.run.smithi046.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-22T02:17:36.875 INFO:teuthology.orchestra.run.smithi046.stdout:The following additional packages will be installed: 2024-02-22T02:17:36.876 INFO:teuthology.orchestra.run.smithi046.stdout: linux-generic linux-headers-5.15.0-94 linux-headers-5.15.0-94-generic 2024-02-22T02:17:36.876 INFO:teuthology.orchestra.run.smithi046.stdout: linux-headers-generic linux-image-5.15.0-94-generic 2024-02-22T02:17:36.876 INFO:teuthology.orchestra.run.smithi046.stdout: linux-modules-5.15.0-94-generic linux-modules-extra-5.15.0-94-generic 2024-02-22T02:17:36.877 INFO:teuthology.orchestra.run.smithi046.stdout:Suggested packages: 2024-02-22T02:17:36.877 INFO:teuthology.orchestra.run.smithi046.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-02-22T02:17:36.877 INFO:teuthology.orchestra.run.smithi046.stdout:Recommended packages: 2024-02-22T02:17:36.877 INFO:teuthology.orchestra.run.smithi046.stdout: thermald 2024-02-22T02:17:36.918 INFO:teuthology.orchestra.run.smithi046.stdout:The following NEW packages will be installed: 2024-02-22T02:17:36.918 INFO:teuthology.orchestra.run.smithi046.stdout: linux-headers-5.15.0-94 linux-headers-5.15.0-94-generic 2024-02-22T02:17:36.918 INFO:teuthology.orchestra.run.smithi046.stdout: linux-image-5.15.0-94-generic linux-modules-5.15.0-94-generic 2024-02-22T02:17:36.919 INFO:teuthology.orchestra.run.smithi046.stdout: linux-modules-extra-5.15.0-94-generic 2024-02-22T02:17:36.920 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages will be upgraded: 2024-02-22T02:17:36.921 INFO:teuthology.orchestra.run.smithi046.stdout: linux-generic linux-headers-generic linux-image-generic 2024-02-22T02:17:36.978 INFO:teuthology.orchestra.run.smithi046.stdout:3 upgraded, 5 newly installed, 0 to remove and 287 not upgraded. 2024-02-22T02:17:36.979 INFO:teuthology.orchestra.run.smithi046.stdout:Need to get 113 MB of archives. 2024-02-22T02:17:36.979 INFO:teuthology.orchestra.run.smithi046.stdout:After this operation, 582 MB of additional disk space will be used. 2024-02-22T02:17:36.979 INFO:teuthology.orchestra.run.smithi046.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-94-generic amd64 5.15.0-94.104 [22.6 MB] 2024-02-22T02:17:37.519 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-94-generic amd64 5.15.0-94.104 [11.5 MB] 2024-02-22T02:17:37.692 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-94-generic amd64 5.15.0-94.104 [63.9 MB] 2024-02-22T02:17:38.893 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.94.91 [1,698 B] 2024-02-22T02:17:38.893 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.94.91 [2,458 B] 2024-02-22T02:17:38.979 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-94 all 5.15.0-94.104 [12.4 MB] 2024-02-22T02:17:39.140 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-94-generic amd64 5.15.0-94.104 [2,877 kB] 2024-02-22T02:17:39.172 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.94.91 [2,306 B] 2024-02-22T02:17:39.584 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 113 MB in 2s (50.4 MB/s) 2024-02-22T02:17:39.766 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-modules-5.15.0-94-generic. 2024-02-22T02:17:41.751 INFO:teuthology.orchestra.run.smithi046.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-02-22T02:17:41.756 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../0-linux-modules-5.15.0-94-generic_5.15.0-94.104_amd64.deb ... 2024-02-22T02:17:41.839 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-modules-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-22T02:17:43.906 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-image-5.15.0-94-generic. 2024-02-22T02:17:43.927 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../1-linux-image-5.15.0-94-generic_5.15.0-94.104_amd64.deb ... 2024-02-22T02:17:44.023 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-image-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-22T02:17:44.484 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-94-generic. 2024-02-22T02:17:44.505 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-94-generic_5.15.0-94.104_amd64.deb ... 2024-02-22T02:17:44.542 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-modules-extra-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-22T02:17:51.292 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../3-linux-generic_5.15.0.94.91_amd64.deb ... 2024-02-22T02:17:51.416 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-generic (5.15.0.94.91) over (5.15.0.56.54) ... 2024-02-22T02:17:51.741 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.94.91_amd64.deb ... 2024-02-22T02:17:51.862 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-image-generic (5.15.0.94.91) over (5.15.0.56.54) ... 2024-02-22T02:17:52.148 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-headers-5.15.0-94. 2024-02-22T02:17:52.170 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../5-linux-headers-5.15.0-94_5.15.0-94.104_all.deb ... 2024-02-22T02:17:52.214 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-headers-5.15.0-94 (5.15.0-94.104) ... 2024-02-22T02:17:57.742 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-headers-5.15.0-94-generic. 2024-02-22T02:17:57.778 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../6-linux-headers-5.15.0-94-generic_5.15.0-94.104_amd64.deb ... 2024-02-22T02:17:57.816 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-headers-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-22T02:18:00.564 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.94.91_amd64.deb ... 2024-02-22T02:18:00.684 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-headers-generic (5.15.0.94.91) over (5.15.0.56.54) ... 2024-02-22T02:18:01.044 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-headers-5.15.0-94 (5.15.0-94.104) ... 2024-02-22T02:18:01.147 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-headers-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-22T02:18:01.265 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-headers-generic (5.15.0.94.91) ... 2024-02-22T02:18:01.392 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-modules-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-22T02:18:02.796 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-image-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-22T02:18:04.569 INFO:teuthology.orchestra.run.smithi046.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-02-22T02:18:04.569 INFO:teuthology.orchestra.run.smithi046.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-02-22T02:18:04.569 INFO:teuthology.orchestra.run.smithi046.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-94-generic 2024-02-22T02:18:04.569 INFO:teuthology.orchestra.run.smithi046.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-94-generic 2024-02-22T02:18:04.725 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-modules-extra-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-22T02:18:06.216 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-image-generic (5.15.0.94.91) ... 2024-02-22T02:18:06.340 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-generic (5.15.0.94.91) ... 2024-02-22T02:18:06.467 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for linux-image-5.15.0-94-generic (5.15.0-94.104) ... 2024-02-22T02:18:06.527 INFO:teuthology.orchestra.run.smithi046.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-02-22T02:18:06.527 INFO:teuthology.orchestra.run.smithi046.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-94-generic 2024-02-22T02:18:20.902 INFO:teuthology.orchestra.run.smithi046.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-02-22T02:18:20.902 INFO:teuthology.orchestra.run.smithi046.stdout:Sourcing file `/etc/default/grub' 2024-02-22T02:18:20.911 INFO:teuthology.orchestra.run.smithi046.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-22T02:18:20.962 INFO:teuthology.orchestra.run.smithi046.stdout:Generating grub configuration file ... 2024-02-22T02:18:21.303 INFO:teuthology.orchestra.run.smithi046.stdout:Found linux image: /boot/vmlinuz-5.15.0-94-generic 2024-02-22T02:18:21.338 INFO:teuthology.orchestra.run.smithi046.stdout:Found initrd image: /boot/initrd.img-5.15.0-94-generic 2024-02-22T02:18:21.697 INFO:teuthology.orchestra.run.smithi046.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-22T02:18:21.704 INFO:teuthology.orchestra.run.smithi046.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-22T02:18:21.774 INFO:teuthology.orchestra.run.smithi046.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-22T02:18:21.780 INFO:teuthology.orchestra.run.smithi046.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-22T02:18:21.937 INFO:teuthology.orchestra.run.smithi046.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-22T02:18:21.937 INFO:teuthology.orchestra.run.smithi046.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-02-22T02:18:21.938 INFO:teuthology.orchestra.run.smithi046.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-22T02:18:21.983 INFO:teuthology.orchestra.run.smithi046.stdout:done 2024-02-22T02:18:22.432 DEBUG:teuthology.orchestra.run.smithi046:> dpkg -s linux-image-generic 2024-02-22T02:18:22.448 INFO:teuthology.orchestra.run.smithi046.stdout:Package: linux-image-generic 2024-02-22T02:18:22.448 INFO:teuthology.orchestra.run.smithi046.stdout:Status: install ok installed 2024-02-22T02:18:22.448 INFO:teuthology.orchestra.run.smithi046.stdout:Priority: optional 2024-02-22T02:18:22.448 INFO:teuthology.orchestra.run.smithi046.stdout:Section: kernel 2024-02-22T02:18:22.449 INFO:teuthology.orchestra.run.smithi046.stdout:Installed-Size: 21 2024-02-22T02:18:22.449 INFO:teuthology.orchestra.run.smithi046.stdout:Maintainer: Ubuntu Kernel Team 2024-02-22T02:18:22.449 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: amd64 2024-02-22T02:18:22.449 INFO:teuthology.orchestra.run.smithi046.stdout:Source: linux-meta 2024-02-22T02:18:22.449 INFO:teuthology.orchestra.run.smithi046.stdout:Version: 5.15.0.94.91 2024-02-22T02:18:22.449 INFO:teuthology.orchestra.run.smithi046.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-94), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-02-22T02:18:22.449 INFO:teuthology.orchestra.run.smithi046.stdout:Depends: linux-image-5.15.0-94-generic, linux-modules-extra-5.15.0-94-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-22T02:18:22.449 INFO:teuthology.orchestra.run.smithi046.stdout:Recommends: thermald 2024-02-22T02:18:22.449 INFO:teuthology.orchestra.run.smithi046.stdout:Description: Generic Linux kernel image 2024-02-22T02:18:22.449 INFO:teuthology.orchestra.run.smithi046.stdout: This package will always depend on the latest generic kernel image 2024-02-22T02:18:22.449 INFO:teuthology.orchestra.run.smithi046.stdout: available. 2024-02-22T02:18:22.450 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-94-generic 2024-02-22T02:18:22.450 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-22T02:18:22.450 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-22T02:18:22.450 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-22T02:18:22.451 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2024-02-22T02:18:22.451 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get clean 2024-02-22T02:18:22.623 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get update 2024-02-22T02:18:22.748 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-02-22T02:18:22.749 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-22T02:18:22.764 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-22T02:18:22.785 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-22T02:18:24.075 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-02-22T02:18:24.093 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-22T02:18:24.154 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-02-22T02:18:24.351 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2024-02-22T02:18:24.352 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2024-02-22T02:18:24.531 INFO:teuthology.orchestra.run.smithi046.stdout:linux-image-generic is already the newest version (5.15.0.94.91). 2024-02-22T02:18:24.531 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages were automatically installed and are no longer required: 2024-02-22T02:18:24.531 INFO:teuthology.orchestra.run.smithi046.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-22T02:18:24.531 INFO:teuthology.orchestra.run.smithi046.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-22T02:18:24.531 INFO:teuthology.orchestra.run.smithi046.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-22T02:18:24.532 INFO:teuthology.orchestra.run.smithi046.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-22T02:18:24.532 INFO:teuthology.orchestra.run.smithi046.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-22T02:18:24.532 INFO:teuthology.orchestra.run.smithi046.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-22T02:18:24.574 INFO:teuthology.orchestra.run.smithi046.stdout:0 upgraded, 0 newly installed, 0 to remove and 287 not upgraded. 2024-02-22T02:18:24.576 DEBUG:teuthology.orchestra.run.smithi046:> dpkg -s linux-image-generic 2024-02-22T02:18:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:Package: linux-image-generic 2024-02-22T02:18:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:Status: install ok installed 2024-02-22T02:18:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:Priority: optional 2024-02-22T02:18:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:Section: kernel 2024-02-22T02:18:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:Installed-Size: 21 2024-02-22T02:18:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:Maintainer: Ubuntu Kernel Team 2024-02-22T02:18:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: amd64 2024-02-22T02:18:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:Source: linux-meta 2024-02-22T02:18:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:Version: 5.15.0.94.91 2024-02-22T02:18:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.2), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-94), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.2) 2024-02-22T02:18:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:Depends: linux-image-5.15.0-94-generic, linux-modules-extra-5.15.0-94-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-22T02:18:24.594 INFO:teuthology.orchestra.run.smithi046.stdout:Recommends: thermald 2024-02-22T02:18:24.594 INFO:teuthology.orchestra.run.smithi046.stdout:Description: Generic Linux kernel image 2024-02-22T02:18:24.594 INFO:teuthology.orchestra.run.smithi046.stdout: This package will always depend on the latest generic kernel image 2024-02-22T02:18:24.594 INFO:teuthology.orchestra.run.smithi046.stdout: available. 2024-02-22T02:18:24.594 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-02-22T02:18:24.639 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.HwLt4w40ZR 2024-02-22T02:18:24.639 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub/grub.cfg /tmp/tmp.HwLt4w40ZR 2024-02-22T02:18:24.695 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.HwLt4w40ZR 2024-02-22T02:18:24.822 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.HwLt4w40ZR is 10KB 2024-02-22T02:18:24.835 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.HwLt4w40ZR 2024-02-22T02:18:24.841 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-02-22T02:18:24.895 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-02-22T02:18:24.896 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-02-22T02:18:24.896 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-02-22T02:18:24.962 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-94-generic 2024-02-22T02:18:24.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo update-grub 2024-02-22T02:18:26.172 INFO:teuthology.orchestra.run.smithi046.stderr:Sourcing file `/etc/default/grub' 2024-02-22T02:18:26.173 INFO:teuthology.orchestra.run.smithi046.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-22T02:18:26.186 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2024-02-22T02:18:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.15.0-94-generic 2024-02-22T02:18:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.15.0-94-generic 2024-02-22T02:18:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-22T02:18:26.729 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-22T02:18:26.774 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-22T02:18:26.780 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-22T02:18:26.919 INFO:teuthology.orchestra.run.smithi046.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-22T02:18:26.919 INFO:teuthology.orchestra.run.smithi046.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-02-22T02:18:26.919 INFO:teuthology.orchestra.run.smithi046.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-22T02:18:26.936 INFO:teuthology.orchestra.run.smithi046.stderr:done 2024-02-22T02:18:26.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2024-02-22T02:18:56.956 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-22T02:18:56.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-02-22T02:18:56.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-22T02:19:15.388 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-02-22T02:19:24.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-02-22T02:19:24.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-22T02:19:27.452 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-02-22T02:19:39.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-02-22T02:19:39.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-22T02:20:39.521 DEBUG:teuthology.orchestra.remote:timed out 2024-02-22T02:20:54.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-02-22T02:20:54.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-22T02:20:54.806 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-02-22T02:20:55.445 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-02-22T02:20:55.445 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-94-generic"... 2024-02-22T02:20:55.446 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-02-22T02:20:55.491 INFO:teuthology.orchestra.run.smithi046.stdout:5.15.0-94-generic 2024-02-22T02:20:55.491 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-94-generic vs 5.15.0-94-generic 2024-02-22T02:20:55.491 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-22T02:20:55.492 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-02-22T02:20:56.493 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-22T02:20:56.493 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-22T02:20:56.560 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2024-02-22T02:20:56.608 DEBUG:teuthology.parallel:result is None 2024-02-22T02:20:56.608 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-22T02:20:56.615 INFO:teuthology.task.internal:Creating test directory... 2024-02-22T02:20:56.615 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-22T02:20:56.621 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-22T02:20:56.634 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-22T02:20:56.644 INFO:teuthology.task.internal:Creating archive directory... 2024-02-22T02:20:56.645 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-22T02:20:56.676 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-22T02:20:56.682 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-22T02:20:56.682 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-22T02:20:56.735 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:20:56.744 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:20:56.746 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-22T02:20:56.753 INFO:teuthology.task.internal:Configuring sudo... 2024-02-22T02:20:56.753 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-22T02:20:56.804 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-22T02:20:56.813 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-22T02:20:56.813 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-22T02:20:56.851 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-22T02:20:56.899 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-22T02:20:56.948 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-02-22T02:20:56.948 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-22T02:20:57.014 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-02-22T02:20:57.085 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-22T02:20:57.094 INFO:teuthology.task.internal:Starting timer... 2024-02-22T02:20:57.094 INFO:teuthology.run_tasks:Running task pcp... 2024-02-22T02:20:57.103 INFO:teuthology.run_tasks:Running task selinux... 2024-02-22T02:20:57.111 DEBUG:teuthology.task.selinux:Excluding smithi046: OS 'ubuntu' does not support SELinux 2024-02-22T02:20:57.111 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-22T02:20:57.111 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-02-22T02:20:57.111 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-22T02:20:57.111 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-22T02:20:57.120 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-22T02:20:57.137 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-22T02:20:57.139 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-22T02:28:03.930 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com')] 2024-02-22T02:28:03.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-02-22T02:28:03.933 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-22T02:28:04.016 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-02-22T02:28:04.100 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-02-22T02:28:04.100 INFO:teuthology.run_tasks:Running task clock... 2024-02-22T02:28:04.111 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-22T02:28:04.111 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-22T02:28:04.112 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-22T02:28:04.168 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-02-22T02:28:04.168 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: Command line: ntpd -gq 2024-02-22T02:28:04.168 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: ---------------------------------------------------- 2024-02-22T02:28:04.168 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: ntp-4 is maintained by Network Time Foundation, 2024-02-22T02:28:04.168 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-02-22T02:28:04.168 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: corporation. Support and training for ntp-4 are 2024-02-22T02:28:04.168 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: available at https://www.nwtime.org/support 2024-02-22T02:28:04.168 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: ---------------------------------------------------- 2024-02-22T02:28:04.169 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: proto: precision = 0.061 usec (-24) 2024-02-22T02:28:04.169 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: basedate set to 2022-02-04 2024-02-22T02:28:04.169 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: gps base set to 2022-02-06 (week 2196) 2024-02-22T02:28:04.169 INFO:teuthology.orchestra.run.smithi046.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-22T02:28:04.170 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-22T02:28:04.170 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: restrict ::: KOD does nothing without LIMITED. 2024-02-22T02:28:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:restrict ::: KOD does nothing without LIMITED. 2024-02-22T02:28:04.171 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: Listen and drop on 0 v6wildcard [::]:123 2024-02-22T02:28:04.171 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-02-22T02:28:04.172 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: Listen normally on 2 lo 127.0.0.1:123 2024-02-22T02:28:04.172 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: Listen normally on 3 ens1f0 172.21.15.46:123 2024-02-22T02:28:04.172 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: Listen normally on 4 lo [::1]:123 2024-02-22T02:28:04.172 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15f8%4]:123 2024-02-22T02:28:04.172 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:04 ntpd[17081]: Listening on routing socket on fd #22 for interface updates 2024-02-22T02:28:05.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:05.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:05.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:05.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:05.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:05.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:05.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:05.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:05.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-22T02:28:05.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: can't open /var/log/ntpstats/peerstats.20240222: Permission denied 2024-02-22T02:28:05.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:05.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:05.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-22T02:28:05.170 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:05 ntpd[17081]: can't open /var/log/ntpstats/peerstats.20240222: Permission denied 2024-02-22T02:28:07.171 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:07 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:07.171 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:07 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:07.171 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:07 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:07.171 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:07 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:07.171 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:07 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:07.171 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:07 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:07.171 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:07 ntpd[17081]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-22T02:28:07.172 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:07 ntpd[17081]: can't open /var/log/ntpstats/peerstats.20240222: Permission denied 2024-02-22T02:28:07.172 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:07 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:07.172 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:07 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:07.172 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:07 ntpd[17081]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-22T02:28:07.172 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:07 ntpd[17081]: can't open /var/log/ntpstats/peerstats.20240222: Permission denied 2024-02-22T02:28:08.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:08 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:08.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:08 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:08.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:08 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:08.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:08 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:08.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:08 ntpd[17081]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-22T02:28:08.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:08 ntpd[17081]: can't open /var/log/ntpstats/peerstats.20240222: Permission denied 2024-02-22T02:28:09.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:09 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:09.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:09 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:09.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:09 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:09.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:09 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:09.170 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:09 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:09.170 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:09 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:09.170 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:09 ntpd[17081]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-22T02:28:09.170 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:09 ntpd[17081]: can't open /var/log/ntpstats/peerstats.20240222: Permission denied 2024-02-22T02:28:09.170 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:09 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:09.170 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:09 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:09.170 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:09 ntpd[17081]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-22T02:28:09.170 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:09 ntpd[17081]: can't open /var/log/ntpstats/peerstats.20240222: Permission denied 2024-02-22T02:28:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:10 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:10 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:10.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:10 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:10.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:10 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:10.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:10 ntpd[17081]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-22T02:28:10.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:10 ntpd[17081]: can't open /var/log/ntpstats/peerstats.20240222: Permission denied 2024-02-22T02:28:11.168 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:11 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:11.168 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:11 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:11 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:11 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:11 ntpd[17081]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-22T02:28:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:11 ntpd[17081]: can't open /var/log/ntpstats/rawstats.20240222: Permission denied 2024-02-22T02:28:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:11 ntpd[17081]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-22T02:28:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:11 ntpd[17081]: can't open /var/log/ntpstats/peerstats.20240222: Permission denied 2024-02-22T02:28:11.169 INFO:teuthology.orchestra.run.smithi046.stdout:22 Feb 02:28:11 ntpd[17081]: ntpd: time slew -0.001695 s 2024-02-22T02:28:11.169 INFO:teuthology.orchestra.run.smithi046.stdout:ntpd: time slew -0.001695s 2024-02-22T02:28:11.170 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:11 ntpd[17081]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-02-22T02:28:11.170 INFO:teuthology.orchestra.run.smithi046.stderr:22 Feb 02:28:11 ntpd[17081]: can't open /var/log/ntpstats/loopstats.20240222: Permission denied 2024-02-22T02:28:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: remote refid st t when poll reach delay offset jitter 2024-02-22T02:28:11.218 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================== 2024-02-22T02:28:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-22T02:28:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-22T02:28:11.218 INFO:teuthology.orchestra.run.smithi046.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-22T02:28:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-22T02:28:11.219 INFO:teuthology.run_tasks:Running task install... 2024-02-22T02:28:11.228 DEBUG:teuthology.task.install:project ceph 2024-02-22T02:28:11.229 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'dbe7fcec16eddee6239e4dd68c4d0203c7df6461'}} 2024-02-22T02:28:11.229 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'dbe7fcec16eddee6239e4dd68c4d0203c7df6461'} 2024-02-22T02:28:11.229 INFO:teuthology.task.install:Using flavor: default 2024-02-22T02:28:11.235 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-22T02:28:11.235 INFO:teuthology.task.install:extra packages: [] 2024-02-22T02:28:11.235 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-key list | grep Ceph 2024-02-22T02:28:11.327 INFO:teuthology.orchestra.run.smithi046.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-02-22T02:28:11.359 INFO:teuthology.orchestra.run.smithi046.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-02-22T02:28:11.359 INFO:teuthology.orchestra.run.smithi046.stdout:uid [ unknown] Ceph.com (release key) 2024-02-22T02:28:11.359 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-02-22T02:28:11.359 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=dbe7fcec16eddee6239e4dd68c4d0203c7df6461 2024-02-22T02:28:11.500 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default/ 2024-02-22T02:28:11.500 INFO:teuthology.task.install.deb:Package version is 19.0.0-1633-gdbe7fcec-1jammy 2024-02-22T02:28:11.824 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-02-22T02:28:11.824 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-02-22T02:28:11.839 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get update 2024-02-22T02:28:12.016 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-22T02:28:12.016 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-02-22T02:28:12.029 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-22T02:28:12.049 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-22T02:28:12.348 INFO:teuthology.orchestra.run.smithi046.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy InRelease 2024-02-22T02:28:12.441 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-02-22T02:28:12.537 INFO:teuthology.orchestra.run.smithi046.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy Release.gpg 2024-02-22T02:28:12.632 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-02-22T02:28:12.739 INFO:teuthology.orchestra.run.smithi046.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2024-02-22T02:28:12.833 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 139 kB in 1s (175 kB/s) 2024-02-22T02:28:13.737 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-02-22T02:28:13.752 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1633-gdbe7fcec-1jammy cephadm=19.0.0-1633-gdbe7fcec-1jammy ceph-mds=19.0.0-1633-gdbe7fcec-1jammy ceph-mgr=19.0.0-1633-gdbe7fcec-1jammy ceph-common=19.0.0-1633-gdbe7fcec-1jammy ceph-fuse=19.0.0-1633-gdbe7fcec-1jammy ceph-test=19.0.0-1633-gdbe7fcec-1jammy ceph-volume=19.0.0-1633-gdbe7fcec-1jammy radosgw=19.0.0-1633-gdbe7fcec-1jammy python3-rados=19.0.0-1633-gdbe7fcec-1jammy python3-rgw=19.0.0-1633-gdbe7fcec-1jammy python3-cephfs=19.0.0-1633-gdbe7fcec-1jammy python3-rbd=19.0.0-1633-gdbe7fcec-1jammy libcephfs2=19.0.0-1633-gdbe7fcec-1jammy libcephfs-dev=19.0.0-1633-gdbe7fcec-1jammy librados2=19.0.0-1633-gdbe7fcec-1jammy librbd1=19.0.0-1633-gdbe7fcec-1jammy rbd-fuse=19.0.0-1633-gdbe7fcec-1jammy 2024-02-22T02:28:13.803 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2024-02-22T02:28:13.977 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2024-02-22T02:28:13.977 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2024-02-22T02:28:14.142 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages were automatically installed and are no longer required: 2024-02-22T02:28:14.142 INFO:teuthology.orchestra.run.smithi046.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-22T02:28:14.142 INFO:teuthology.orchestra.run.smithi046.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-22T02:28:14.142 INFO:teuthology.orchestra.run.smithi046.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-22T02:28:14.142 INFO:teuthology.orchestra.run.smithi046.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-22T02:28:14.143 INFO:teuthology.orchestra.run.smithi046.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-22T02:28:14.143 INFO:teuthology.orchestra.run.smithi046.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-22T02:28:14.143 INFO:teuthology.orchestra.run.smithi046.stdout:The following additional packages will be installed: 2024-02-22T02:28:14.143 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-02-22T02:28:14.143 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-02-22T02:28:14.145 INFO:teuthology.orchestra.run.smithi046.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-02-22T02:28:14.145 INFO:teuthology.orchestra.run.smithi046.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-02-22T02:28:14.145 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-02-22T02:28:14.145 INFO:teuthology.orchestra.run.smithi046.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-02-22T02:28:14.145 INFO:teuthology.orchestra.run.smithi046.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-02-22T02:28:14.145 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-02-22T02:28:14.145 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-02-22T02:28:14.145 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-02-22T02:28:14.145 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-22T02:28:14.145 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-22T02:28:14.146 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-22T02:28:14.146 INFO:teuthology.orchestra.run.smithi046.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-22T02:28:14.146 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-22T02:28:14.146 INFO:teuthology.orchestra.run.smithi046.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-22T02:28:14.146 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-02-22T02:28:14.146 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-02-22T02:28:14.146 INFO:teuthology.orchestra.run.smithi046.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-02-22T02:28:14.146 INFO:teuthology.orchestra.run.smithi046.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-02-22T02:28:14.146 INFO:teuthology.orchestra.run.smithi046.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-02-22T02:28:14.146 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-02-22T02:28:14.146 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-02-22T02:28:14.146 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet zip 2024-02-22T02:28:14.147 INFO:teuthology.orchestra.run.smithi046.stdout:Suggested packages: 2024-02-22T02:28:14.147 INFO:teuthology.orchestra.run.smithi046.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-02-22T02:28:14.147 INFO:teuthology.orchestra.run.smithi046.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-02-22T02:28:14.147 INFO:teuthology.orchestra.run.smithi046.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-02-22T02:28:14.147 INFO:teuthology.orchestra.run.smithi046.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-02-22T02:28:14.147 INFO:teuthology.orchestra.run.smithi046.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-02-22T02:28:14.147 INFO:teuthology.orchestra.run.smithi046.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-02-22T02:28:14.147 INFO:teuthology.orchestra.run.smithi046.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-02-22T02:28:14.147 INFO:teuthology.orchestra.run.smithi046.stdout:Recommended packages: 2024-02-22T02:28:14.147 INFO:teuthology.orchestra.run.smithi046.stdout: btrfs-tools 2024-02-22T02:28:14.249 INFO:teuthology.orchestra.run.smithi046.stdout:The following NEW packages will be installed: 2024-02-22T02:28:14.249 INFO:teuthology.orchestra.run.smithi046.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-02-22T02:28:14.249 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-02-22T02:28:14.249 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-02-22T02:28:14.249 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-22T02:28:14.250 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-02-22T02:28:14.251 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-02-22T02:28:14.251 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-02-22T02:28:14.251 INFO:teuthology.orchestra.run.smithi046.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-02-22T02:28:14.251 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-02-22T02:28:14.251 INFO:teuthology.orchestra.run.smithi046.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-02-22T02:28:14.251 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-02-22T02:28:14.251 INFO:teuthology.orchestra.run.smithi046.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-02-22T02:28:14.437 INFO:teuthology.orchestra.run.smithi046.stdout:0 upgraded, 118 newly installed, 0 to remove and 274 not upgraded. 2024-02-22T02:28:14.437 INFO:teuthology.orchestra.run.smithi046.stdout:Need to get 176 MB of archives. 2024-02-22T02:28:14.437 INFO:teuthology.orchestra.run.smithi046.stdout:After this operation, 730 MB of additional disk space will be used. 2024-02-22T02:28:14.437 INFO:teuthology.orchestra.run.smithi046.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-02-22T02:28:14.668 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1633-gdbe7fcec-1jammy [3,534 kB] 2024-02-22T02:28:14.863 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-02-22T02:28:14.875 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-02-22T02:28:14.957 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-02-22T02:28:15.187 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-02-22T02:28:15.199 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-02-22T02:28:15.228 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-02-22T02:28:15.237 INFO:teuthology.orchestra.run.smithi046.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-02-22T02:28:15.239 INFO:teuthology.orchestra.run.smithi046.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-02-22T02:28:15.240 INFO:teuthology.orchestra.run.smithi046.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-02-22T02:28:15.258 INFO:teuthology.orchestra.run.smithi046.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-02-22T02:28:15.426 INFO:teuthology.orchestra.run.smithi046.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-02-22T02:28:15.835 INFO:teuthology.orchestra.run.smithi046.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-02-22T02:28:15.868 INFO:teuthology.orchestra.run.smithi046.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-02-22T02:28:15.870 INFO:teuthology.orchestra.run.smithi046.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-02-22T02:28:15.950 INFO:teuthology.orchestra.run.smithi046.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-02-22T02:28:16.002 INFO:teuthology.orchestra.run.smithi046.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-02-22T02:28:16.023 INFO:teuthology.orchestra.run.smithi046.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-02-22T02:28:16.030 INFO:teuthology.orchestra.run.smithi046.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-02-22T02:28:16.031 INFO:teuthology.orchestra.run.smithi046.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-02-22T02:28:16.043 INFO:teuthology.orchestra.run.smithi046.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-02-22T02:28:16.044 INFO:teuthology.orchestra.run.smithi046.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-02-22T02:28:16.203 INFO:teuthology.orchestra.run.smithi046.stdout:Get:24 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1633-gdbe7fcec-1jammy [3,218 kB] 2024-02-22T02:28:16.208 INFO:teuthology.orchestra.run.smithi046.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-02-22T02:28:16.292 INFO:teuthology.orchestra.run.smithi046.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-02-22T02:28:16.320 INFO:teuthology.orchestra.run.smithi046.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-02-22T02:28:16.337 INFO:teuthology.orchestra.run.smithi046.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-02-22T02:28:16.381 INFO:teuthology.orchestra.run.smithi046.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-02-22T02:28:16.592 INFO:teuthology.orchestra.run.smithi046.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-02-22T02:28:16.607 INFO:teuthology.orchestra.run.smithi046.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-02-22T02:28:16.615 INFO:teuthology.orchestra.run.smithi046.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-02-22T02:28:16.619 INFO:teuthology.orchestra.run.smithi046.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-02-22T02:28:16.640 INFO:teuthology.orchestra.run.smithi046.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-02-22T02:28:16.644 INFO:teuthology.orchestra.run.smithi046.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-02-22T02:28:16.810 INFO:teuthology.orchestra.run.smithi046.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-02-22T02:28:17.142 INFO:teuthology.orchestra.run.smithi046.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-02-22T02:28:17.174 INFO:teuthology.orchestra.run.smithi046.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-02-22T02:28:17.182 INFO:teuthology.orchestra.run.smithi046.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1633-gdbe7fcec-1jammy [710 kB] 2024-02-22T02:28:17.184 INFO:teuthology.orchestra.run.smithi046.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-02-22T02:28:17.273 INFO:teuthology.orchestra.run.smithi046.stdout:Get:41 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1633-gdbe7fcec-1jammy [356 kB] 2024-02-22T02:28:17.330 INFO:teuthology.orchestra.run.smithi046.stdout:Get:42 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1633-gdbe7fcec-1jammy [32.8 kB] 2024-02-22T02:28:17.332 INFO:teuthology.orchestra.run.smithi046.stdout:Get:43 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1633-gdbe7fcec-1jammy [180 kB] 2024-02-22T02:28:17.360 INFO:teuthology.orchestra.run.smithi046.stdout:Get:44 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1633-gdbe7fcec-1jammy [64.8 kB] 2024-02-22T02:28:17.362 INFO:teuthology.orchestra.run.smithi046.stdout:Get:45 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1633-gdbe7fcec-1jammy [330 kB] 2024-02-22T02:28:17.386 INFO:teuthology.orchestra.run.smithi046.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-02-22T02:28:17.387 INFO:teuthology.orchestra.run.smithi046.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-02-22T02:28:17.394 INFO:teuthology.orchestra.run.smithi046.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-02-22T02:28:17.400 INFO:teuthology.orchestra.run.smithi046.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-02-22T02:28:17.415 INFO:teuthology.orchestra.run.smithi046.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-02-22T02:28:17.419 INFO:teuthology.orchestra.run.smithi046.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1633-gdbe7fcec-1jammy [6,575 kB] 2024-02-22T02:28:17.436 INFO:teuthology.orchestra.run.smithi046.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-02-22T02:28:17.442 INFO:teuthology.orchestra.run.smithi046.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-02-22T02:28:17.620 INFO:teuthology.orchestra.run.smithi046.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-02-22T02:28:17.882 INFO:teuthology.orchestra.run.smithi046.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-02-22T02:28:18.060 INFO:teuthology.orchestra.run.smithi046.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-02-22T02:28:18.074 INFO:teuthology.orchestra.run.smithi046.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-02-22T02:28:18.094 INFO:teuthology.orchestra.run.smithi046.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-02-22T02:28:18.158 INFO:teuthology.orchestra.run.smithi046.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-02-22T02:28:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1633-gdbe7fcec-1jammy [112 kB] 2024-02-22T02:28:18.403 INFO:teuthology.orchestra.run.smithi046.stdout:Get:61 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1633-gdbe7fcec-1jammy [467 kB] 2024-02-22T02:28:18.446 INFO:teuthology.orchestra.run.smithi046.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-02-22T02:28:18.449 INFO:teuthology.orchestra.run.smithi046.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-02-22T02:28:18.453 INFO:teuthology.orchestra.run.smithi046.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-02-22T02:28:18.453 INFO:teuthology.orchestra.run.smithi046.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-02-22T02:28:18.457 INFO:teuthology.orchestra.run.smithi046.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-02-22T02:28:18.481 INFO:teuthology.orchestra.run.smithi046.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1633-gdbe7fcec-1jammy [24.2 MB] 2024-02-22T02:28:18.676 INFO:teuthology.orchestra.run.smithi046.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-02-22T02:28:19.114 INFO:teuthology.orchestra.run.smithi046.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-02-22T02:28:19.120 INFO:teuthology.orchestra.run.smithi046.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-02-22T02:28:19.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-02-22T02:28:19.411 INFO:teuthology.orchestra.run.smithi046.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-02-22T02:28:19.413 INFO:teuthology.orchestra.run.smithi046.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-02-22T02:28:19.415 INFO:teuthology.orchestra.run.smithi046.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-02-22T02:28:19.416 INFO:teuthology.orchestra.run.smithi046.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-02-22T02:28:19.418 INFO:teuthology.orchestra.run.smithi046.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-02-22T02:28:19.419 INFO:teuthology.orchestra.run.smithi046.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-02-22T02:28:19.421 INFO:teuthology.orchestra.run.smithi046.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-02-22T02:28:19.603 INFO:teuthology.orchestra.run.smithi046.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-02-22T02:28:20.009 INFO:teuthology.orchestra.run.smithi046.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-02-22T02:28:20.056 INFO:teuthology.orchestra.run.smithi046.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-02-22T02:28:20.076 INFO:teuthology.orchestra.run.smithi046.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-02-22T02:28:20.187 INFO:teuthology.orchestra.run.smithi046.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-02-22T02:28:20.227 INFO:teuthology.orchestra.run.smithi046.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-02-22T02:28:20.238 INFO:teuthology.orchestra.run.smithi046.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-02-22T02:28:20.244 INFO:teuthology.orchestra.run.smithi046.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-02-22T02:28:20.291 INFO:teuthology.orchestra.run.smithi046.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-02-22T02:28:20.294 INFO:teuthology.orchestra.run.smithi046.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-02-22T02:28:20.316 INFO:teuthology.orchestra.run.smithi046.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-02-22T02:28:20.480 INFO:teuthology.orchestra.run.smithi046.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-02-22T02:28:20.753 INFO:teuthology.orchestra.run.smithi046.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-02-22T02:28:20.770 INFO:teuthology.orchestra.run.smithi046.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-02-22T02:28:20.923 INFO:teuthology.orchestra.run.smithi046.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-02-22T02:28:20.952 INFO:teuthology.orchestra.run.smithi046.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-02-22T02:28:21.109 INFO:teuthology.orchestra.run.smithi046.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-02-22T02:28:21.112 INFO:teuthology.orchestra.run.smithi046.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-02-22T02:28:21.114 INFO:teuthology.orchestra.run.smithi046.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-02-22T02:28:21.136 INFO:teuthology.orchestra.run.smithi046.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-02-22T02:28:21.141 INFO:teuthology.orchestra.run.smithi046.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-02-22T02:28:21.353 INFO:teuthology.orchestra.run.smithi046.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1633-gdbe7fcec-1jammy [5,076 kB] 2024-02-22T02:28:21.701 INFO:teuthology.orchestra.run.smithi046.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1633-gdbe7fcec-1jammy [243 kB] 2024-02-22T02:28:21.709 INFO:teuthology.orchestra.run.smithi046.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1633-gdbe7fcec-1jammy [124 kB] 2024-02-22T02:28:21.727 INFO:teuthology.orchestra.run.smithi046.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1633-gdbe7fcec-1jammy [1,540 kB] 2024-02-22T02:28:21.833 INFO:teuthology.orchestra.run.smithi046.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1633-gdbe7fcec-1jammy [6,237 kB] 2024-02-22T02:28:22.271 INFO:teuthology.orchestra.run.smithi046.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1633-gdbe7fcec-1jammy [22.7 MB] 2024-02-22T02:28:23.833 INFO:teuthology.orchestra.run.smithi046.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1633-gdbe7fcec-1jammy [14.1 kB] 2024-02-22T02:28:23.834 INFO:teuthology.orchestra.run.smithi046.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1633-gdbe7fcec-1jammy [866 kB] 2024-02-22T02:28:23.857 INFO:teuthology.orchestra.run.smithi046.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1633-gdbe7fcec-1jammy [2,191 kB] 2024-02-22T02:28:23.985 INFO:teuthology.orchestra.run.smithi046.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1633-gdbe7fcec-1jammy [776 kB] 2024-02-22T02:28:24.026 INFO:teuthology.orchestra.run.smithi046.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1633-gdbe7fcec-1jammy [144 kB] 2024-02-22T02:28:24.042 INFO:teuthology.orchestra.run.smithi046.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1633-gdbe7fcec-1jammy [1,922 kB] 2024-02-22T02:28:24.148 INFO:teuthology.orchestra.run.smithi046.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1633-gdbe7fcec-1jammy [8,625 kB] 2024-02-22T02:28:24.624 INFO:teuthology.orchestra.run.smithi046.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1633-gdbe7fcec-1jammy [14.2 kB] 2024-02-22T02:28:24.624 INFO:teuthology.orchestra.run.smithi046.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1633-gdbe7fcec-1jammy [49.9 MB] 2024-02-22T02:28:27.646 INFO:teuthology.orchestra.run.smithi046.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1633-gdbe7fcec-1jammy [121 kB] 2024-02-22T02:28:27.648 INFO:teuthology.orchestra.run.smithi046.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1633-gdbe7fcec-1jammy [38.4 kB] 2024-02-22T02:28:27.650 INFO:teuthology.orchestra.run.smithi046.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1633-gdbe7fcec-1jammy [13.0 MB] 2024-02-22T02:28:28.199 INFO:teuthology.orchestra.run.smithi046.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-pdhange-testing1/dbe7fcec16eddee6239e4dd68c4d0203c7df6461/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1633-gdbe7fcec-1jammy [91.5 kB] 2024-02-22T02:28:28.777 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 176 MB in 14s (12.6 MB/s) 2024-02-22T02:28:28.953 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-02-22T02:28:29.055 INFO:teuthology.orchestra.run.smithi046.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165994 files and directories currently installed.) 2024-02-22T02:28:29.061 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-02-22T02:28:29.143 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-22T02:28:29.457 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-02-22T02:28:29.487 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-02-22T02:28:29.521 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-22T02:28:29.849 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-02-22T02:28:29.876 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-02-22T02:28:29.916 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-22T02:28:30.370 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-02-22T02:28:30.396 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-22T02:28:30.444 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-22T02:28:30.814 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-02-22T02:28:30.841 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-22T02:28:30.872 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-22T02:28:31.234 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-02-22T02:28:31.261 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-22T02:28:31.300 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-22T02:28:31.662 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-02-22T02:28:31.689 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-02-22T02:28:31.728 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-22T02:28:32.040 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librados2. 2024-02-22T02:28:32.067 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../007-librados2_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:32.098 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librados2 (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:32.559 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librbd1. 2024-02-22T02:28:32.586 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../008-librbd1_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:32.626 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librbd1 (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:33.087 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libcephfs2. 2024-02-22T02:28:33.114 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:33.154 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libcephfs2 (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:33.439 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rados. 2024-02-22T02:28:33.466 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:33.500 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rados (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:34.028 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-02-22T02:28:34.054 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1633-gdbe7fcec-1jammy_all.deb ... 2024-02-22T02:28:34.094 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-ceph-argparse (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:34.347 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cephfs. 2024-02-22T02:28:34.373 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:34.413 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cephfs (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:34.724 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-ceph-common. 2024-02-22T02:28:34.751 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1633-gdbe7fcec-1jammy_all.deb ... 2024-02-22T02:28:34.791 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-ceph-common (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:35.094 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-wcwidth. 2024-02-22T02:28:35.121 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-02-22T02:28:35.152 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-22T02:28:35.447 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-prettytable. 2024-02-22T02:28:35.473 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-02-22T02:28:35.513 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-02-22T02:28:35.816 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rbd. 2024-02-22T02:28:35.832 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:35.874 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rbd (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:36.227 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-02-22T02:28:36.254 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-02-22T02:28:36.294 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-22T02:28:36.538 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-02-22T02:28:36.554 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-02-22T02:28:36.588 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-02-22T02:28:36.883 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-02-22T02:28:36.909 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-02-22T02:28:36.949 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-22T02:28:37.394 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package lua5.1. 2024-02-22T02:28:37.421 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-02-22T02:28:37.461 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-02-22T02:28:37.772 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package lua-any. 2024-02-22T02:28:37.788 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-02-22T02:28:37.830 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking lua-any (27ubuntu1) ... 2024-02-22T02:28:38.083 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package zip. 2024-02-22T02:28:38.098 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-02-22T02:28:38.141 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking zip (3.0-12build2) ... 2024-02-22T02:28:38.428 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package unzip. 2024-02-22T02:28:38.454 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-02-22T02:28:38.486 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-02-22T02:28:38.906 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package luarocks. 2024-02-22T02:28:38.933 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-02-22T02:28:38.973 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-02-22T02:28:39.355 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librgw2. 2024-02-22T02:28:39.382 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../025-librgw2_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:39.422 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librgw2 (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:39.974 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rgw. 2024-02-22T02:28:40.001 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:40.041 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rgw (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:40.370 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liboath0:amd64. 2024-02-22T02:28:40.396 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-02-22T02:28:40.436 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-02-22T02:28:40.764 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libradosstriper1. 2024-02-22T02:28:40.790 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:40.830 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libradosstriper1 (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:41.275 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-common. 2024-02-22T02:28:41.302 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:41.341 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-common (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:42.730 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-base. 2024-02-22T02:28:42.757 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:42.798 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-base (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:43.385 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-02-22T02:28:43.412 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-02-22T02:28:43.451 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-02-22T02:28:43.737 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cheroot. 2024-02-22T02:28:43.764 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-02-22T02:28:43.804 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-22T02:28:44.332 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-02-22T02:28:44.359 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-02-22T02:28:44.398 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-02-22T02:28:44.660 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jaraco.text. 2024-02-22T02:28:44.687 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-02-22T02:28:44.726 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-02-22T02:28:45.029 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-02-22T02:28:45.056 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-02-22T02:28:45.121 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-02-22T02:28:45.423 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-tempora. 2024-02-22T02:28:45.451 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-02-22T02:28:45.490 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-02-22T02:28:45.777 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-portend. 2024-02-22T02:28:45.803 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-02-22T02:28:45.835 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-02-22T02:28:46.113 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-02-22T02:28:46.140 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-02-22T02:28:46.179 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-02-22T02:28:46.482 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cherrypy3. 2024-02-22T02:28:46.509 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-02-22T02:28:46.548 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-02-22T02:28:46.869 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-natsort. 2024-02-22T02:28:46.896 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-02-22T02:28:46.935 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-02-22T02:28:47.947 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-logutils. 2024-02-22T02:28:47.974 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-02-22T02:28:48.014 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-02-22T02:28:48.350 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-markupsafe. 2024-02-22T02:28:48.377 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-02-22T02:28:48.416 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-02-22T02:28:48.686 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-mako. 2024-02-22T02:28:48.702 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-02-22T02:28:48.736 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-22T02:28:49.039 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-simplegeneric. 2024-02-22T02:28:49.067 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-02-22T02:28:49.106 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-02-22T02:28:49.409 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-singledispatch. 2024-02-22T02:28:49.436 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-02-22T02:28:49.475 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-02-22T02:28:49.778 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-webob. 2024-02-22T02:28:49.805 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-02-22T02:28:49.845 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-02-22T02:28:50.198 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-waitress. 2024-02-22T02:28:50.226 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-02-22T02:28:50.267 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-22T02:28:50.559 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-tempita. 2024-02-22T02:28:50.586 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-02-22T02:28:50.617 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-02-22T02:28:50.928 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-paste. 2024-02-22T02:28:50.956 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-02-22T02:28:50.995 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-02-22T02:28:51.348 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-02-22T02:28:51.364 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-02-22T02:28:51.406 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-02-22T02:28:51.675 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pastedeploy. 2024-02-22T02:28:51.699 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-02-22T02:28:51.734 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-02-22T02:28:52.012 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-webtest. 2024-02-22T02:28:52.039 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-02-22T02:28:52.078 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-02-22T02:28:52.373 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pecan. 2024-02-22T02:28:52.400 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-02-22T02:28:52.439 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-02-22T02:28:52.819 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-certifi. 2024-02-22T02:28:52.846 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-02-22T02:28:52.886 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-02-22T02:28:53.222 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-idna. 2024-02-22T02:28:53.249 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-02-22T02:28:53.289 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-idna (3.3-1) ... 2024-02-22T02:28:53.558 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-urllib3. 2024-02-22T02:28:53.585 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-02-22T02:28:53.616 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-22T02:28:53.903 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-requests. 2024-02-22T02:28:53.930 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-02-22T02:28:53.961 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-22T02:28:54.239 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-werkzeug. 2024-02-22T02:28:54.266 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-02-22T02:28:54.330 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-22T02:28:54.767 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-02-22T02:28:54.794 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1633-gdbe7fcec-1jammy_all.deb ... 2024-02-22T02:28:54.825 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:55.214 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-02-22T02:28:55.241 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:55.280 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:55.549 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-bcrypt. 2024-02-22T02:28:55.566 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-02-22T02:28:55.599 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-02-22T02:28:55.886 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-openssl. 2024-02-22T02:28:55.913 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-02-22T02:28:55.952 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-02-22T02:28:56.297 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr. 2024-02-22T02:28:56.324 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:56.363 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:56.725 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mon. 2024-02-22T02:28:56.752 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:56.791 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mon (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:57.387 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-osd. 2024-02-22T02:28:57.414 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:57.453 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-osd (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:58.531 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph. 2024-02-22T02:28:58.547 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../066-ceph_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:58.590 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:58.909 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-fuse. 2024-02-22T02:28:58.936 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:58.975 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-fuse (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:59.337 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mds. 2024-02-22T02:28:59.365 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:59.404 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mds (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:28:59.915 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package cephadm. 2024-02-22T02:28:59.942 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../069-cephadm_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:28:59.973 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking cephadm (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:00.276 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-asyncssh. 2024-02-22T02:29:00.303 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-02-22T02:29:00.342 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-02-22T02:29:00.712 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python-babel-localedata. 2024-02-22T02:29:00.739 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-02-22T02:29:00.778 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-22T02:29:01.833 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-babel. 2024-02-22T02:29:01.861 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-02-22T02:29:01.891 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-02-22T02:29:02.228 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jinja2. 2024-02-22T02:29:02.256 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-02-22T02:29:02.294 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-22T02:29:02.656 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-02-22T02:29:02.683 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1633-gdbe7fcec-1jammy_all.deb ... 2024-02-22T02:29:02.722 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:03.084 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-repoze.lru. 2024-02-22T02:29:03.111 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-02-22T02:29:03.150 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-02-22T02:29:03.453 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-routes. 2024-02-22T02:29:03.470 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-02-22T02:29:03.536 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-02-22T02:29:03.873 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-02-22T02:29:03.890 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1633-gdbe7fcec-1jammy_all.deb ... 2024-02-22T02:29:03.931 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:04.360 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-02-22T02:29:04.387 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-02-22T02:29:04.426 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-22T02:29:04.930 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-joblib. 2024-02-22T02:29:04.957 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-02-22T02:29:04.996 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-02-22T02:29:05.308 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-02-22T02:29:05.324 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-02-22T02:29:05.360 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-02-22T02:29:05.619 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-sklearn. 2024-02-22T02:29:05.646 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-02-22T02:29:05.685 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-22T02:29:06.369 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-02-22T02:29:06.397 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1633-gdbe7fcec-1jammy_all.deb ... 2024-02-22T02:29:06.529 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:07.524 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cachetools. 2024-02-22T02:29:07.552 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-02-22T02:29:07.591 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-02-22T02:29:07.894 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyasn1. 2024-02-22T02:29:07.921 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-02-22T02:29:07.960 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-02-22T02:29:08.297 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-02-22T02:29:08.325 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-02-22T02:29:08.363 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-02-22T02:29:08.791 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rsa. 2024-02-22T02:29:08.819 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-02-22T02:29:08.858 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rsa (4.8-1) ... 2024-02-22T02:29:09.128 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-google-auth. 2024-02-22T02:29:09.156 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-02-22T02:29:09.186 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-02-22T02:29:09.456 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-02-22T02:29:09.484 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-02-22T02:29:09.523 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-22T02:29:09.851 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-websocket. 2024-02-22T02:29:09.879 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-02-22T02:29:10.117 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-02-22T02:29:10.437 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-kubernetes. 2024-02-22T02:29:10.465 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-02-22T02:29:10.504 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-22T02:29:11.167 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-02-22T02:29:11.193 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1633-gdbe7fcec-1jammy_all.deb ... 2024-02-22T02:29:11.234 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:11.589 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libonig5:amd64. 2024-02-22T02:29:11.617 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-02-22T02:29:11.655 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-22T02:29:11.967 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libjq1:amd64. 2024-02-22T02:29:11.984 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-02-22T02:29:12.025 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-22T02:29:12.294 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package jq. 2024-02-22T02:29:12.322 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-02-22T02:29:12.360 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-02-22T02:29:12.613 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package socat. 2024-02-22T02:29:12.641 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-02-22T02:29:12.679 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-02-22T02:29:13.016 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package xmlstarlet. 2024-02-22T02:29:13.033 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-02-22T02:29:13.074 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-02-22T02:29:13.420 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-test. 2024-02-22T02:29:13.436 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:29:13.478 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-test (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:15.716 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-volume. 2024-02-22T02:29:15.733 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1633-gdbe7fcec-1jammy_all.deb ... 2024-02-22T02:29:15.796 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-volume (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:16.111 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libcephfs-dev. 2024-02-22T02:29:16.128 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:29:16.169 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libcephfs-dev (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:16.488 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package lua-socket:amd64. 2024-02-22T02:29:16.516 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-02-22T02:29:16.555 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-22T02:29:16.859 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package lua-sec:amd64. 2024-02-22T02:29:16.888 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-02-22T02:29:16.926 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-02-22T02:29:17.296 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package nvme-cli. 2024-02-22T02:29:17.328 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-02-22T02:29:17.363 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-02-22T02:29:17.751 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package pkg-config. 2024-02-22T02:29:17.780 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-02-22T02:29:17.818 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-02-22T02:29:18.062 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-02-22T02:29:18.079 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-02-22T02:29:18.120 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-02-22T02:29:18.548 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-iniconfig. 2024-02-22T02:29:18.578 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-02-22T02:29:18.615 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-02-22T02:29:18.934 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pastescript. 2024-02-22T02:29:18.963 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-02-22T02:29:19.000 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-02-22T02:29:19.287 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pluggy. 2024-02-22T02:29:19.315 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-02-22T02:29:19.345 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-02-22T02:29:19.640 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-psutil. 2024-02-22T02:29:19.668 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-02-22T02:29:19.706 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-02-22T02:29:20.001 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-py. 2024-02-22T02:29:20.018 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-02-22T02:29:20.059 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-py (1.10.0-1) ... 2024-02-22T02:29:20.362 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pygments. 2024-02-22T02:29:20.391 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-02-22T02:29:20.429 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-02-22T02:29:20.883 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyinotify. 2024-02-22T02:29:20.911 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-02-22T02:29:20.949 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-02-22T02:29:21.268 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-toml. 2024-02-22T02:29:21.297 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-02-22T02:29:21.335 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-02-22T02:29:21.638 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pytest. 2024-02-22T02:29:21.666 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-02-22T02:29:21.704 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-02-22T02:29:22.091 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-simplejson. 2024-02-22T02:29:22.120 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-02-22T02:29:22.157 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-02-22T02:29:22.426 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-02-22T02:29:22.443 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-02-22T02:29:22.485 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-02-22T02:29:23.180 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package radosgw. 2024-02-22T02:29:23.209 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../116-radosgw_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:29:23.238 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking radosgw (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:24.125 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package rbd-fuse. 2024-02-22T02:29:24.141 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1633-gdbe7fcec-1jammy_amd64.deb ... 2024-02-22T02:29:24.182 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking rbd-fuse (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:24.487 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-02-22T02:29:24.746 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-22T02:29:24.872 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-02-22T02:29:25.171 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-02-22T02:29:25.522 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-02-22T02:29:25.980 INFO:teuthology.orchestra.run.smithi046.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-02-22T02:29:26.115 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up cephadm (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:26.573 INFO:teuthology.orchestra.run.smithi046.stdout:Adding system user cephadm....done 2024-02-22T02:29:26.721 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-22T02:29:27.014 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-02-22T02:29:27.282 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-02-22T02:29:27.391 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-02-22T02:29:27.692 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-02-22T02:29:27.960 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-02-22T02:29:28.078 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-py (1.10.0-1) ... 2024-02-22T02:29:28.379 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-02-22T02:29:28.748 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-02-22T02:29:29.007 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-02-22T02:29:29.275 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-02-22T02:29:29.501 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-02-22T02:29:29.736 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-02-22T02:29:29.987 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-02-22T02:29:30.237 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-ceph-argparse (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:30.505 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-22T02:29:30.640 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-22T02:29:30.766 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-02-22T02:29:30.867 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-02-22T02:29:31.101 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-02-22T02:29:31.498 INFO:teuthology.orchestra.run.smithi046.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-02-22T02:29:31.524 INFO:teuthology.orchestra.run.smithi046.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-02-22T02:29:31.587 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-22T02:29:31.713 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-02-22T02:29:32.089 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-22T02:29:32.215 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-02-22T02:29:32.475 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-02-22T02:29:32.726 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-02-22T02:29:33.018 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up zip (3.0-12build2) ... 2024-02-22T02:29:33.144 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-02-22T02:29:33.855 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-02-22T02:29:34.122 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-02-22T02:29:34.283 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-02-22T02:29:34.408 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-22T02:29:34.700 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-02-22T02:29:35.093 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-02-22T02:29:35.361 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-02-22T02:29:35.754 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-22T02:29:36.055 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-22T02:29:36.415 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-02-22T02:29:36.674 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-02-22T02:29:36.792 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-ceph-common (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:37.076 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-idna (3.3-1) ... 2024-02-22T02:29:37.377 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-22T02:29:38.599 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-02-22T02:29:38.817 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-22T02:29:38.943 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-toml (0.10.2-1) ... 2024-02-22T02:29:39.211 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-22T02:29:39.337 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-02-22T02:29:39.463 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-22T02:29:39.772 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-02-22T02:29:40.040 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-02-22T02:29:40.300 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-22T02:29:40.417 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-02-22T02:29:40.710 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-02-22T02:29:41.136 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-02-22T02:29:41.412 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-02-22T02:29:41.672 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-02-22T02:29:41.939 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-02-22T02:29:42.215 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-22T02:29:42.316 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-02-22T02:29:42.584 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-22T02:29:42.685 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-22T02:29:42.961 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-02-22T02:29:43.279 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-02-22T02:29:43.538 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-22T02:29:43.664 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-02-22T02:29:43.798 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-02-22T02:29:43.880 INFO:teuthology.orchestra.run.smithi046.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-02-22T02:29:44.150 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-22T02:29:44.251 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-02-22T02:29:44.619 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-02-22T02:29:44.895 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up lua-any (27ubuntu1) ... 2024-02-22T02:29:45.021 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-portend (3.0.0-1) ... 2024-02-22T02:29:45.280 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-22T02:29:45.415 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-22T02:29:45.741 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-02-22T02:29:46.025 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-22T02:29:46.309 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-02-22T02:29:46.435 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rsa (4.8-1) ... 2024-02-22T02:29:46.728 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-02-22T02:29:46.996 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-22T02:29:47.349 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-02-22T02:29:47.742 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-02-22T02:29:48.027 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-02-22T02:29:48.371 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-22T02:29:48.497 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librados2 (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:48.623 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:48.749 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-02-22T02:29:49.027 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libcephfs2 (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:49.153 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libradosstriper1 (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:49.279 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-02-22T02:29:49.582 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librbd1 (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-modules-core (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:49.849 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-fuse (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:50.076 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-22T02:29:50.076 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-22T02:29:50.550 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libcephfs-dev (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:50.668 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rados (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:50.769 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librgw2 (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:50.895 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-22T02:29:52.157 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rbd (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:52.282 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up rbd-fuse (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:52.400 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rgw (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:52.501 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cephfs (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:52.602 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-common (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:53.153 INFO:teuthology.orchestra.run.smithi046.stdout:Adding group ceph....done 2024-02-22T02:29:53.521 INFO:teuthology.orchestra.run.smithi046.stdout:Adding system user ceph....done 2024-02-22T02:29:53.638 INFO:teuthology.orchestra.run.smithi046.stdout:Setting system user ceph properties....done 2024-02-22T02:29:53.658 INFO:teuthology.orchestra.run.smithi046.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-02-22T02:29:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-02-22T02:29:54.170 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-02-22T02:29:54.636 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-test (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:54.737 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up radosgw (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:55.200 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-22T02:29:55.200 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-22T02:29:55.689 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-base (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:56.077 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-02-22T02:29:56.565 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mds (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:56.802 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-22T02:29:56.802 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-22T02:29:57.317 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:57.542 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-22T02:29:57.542 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-22T02:29:58.035 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-osd (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:58.306 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-22T02:29:58.306 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-22T02:29:58.836 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:58.954 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:59.105 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mon (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:59.322 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-22T02:29:59.322 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-22T02:29:59.814 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-cephadm (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:29:59.966 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:30:00.091 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-dashboard (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:30:00.242 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-volume (19.0.0-1633-gdbe7fcec-1jammy) ... 2024-02-22T02:30:00.612 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-02-22T02:30:00.712 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-02-22T02:30:00.865 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-02-22T02:30:02.066 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-02-22T02:30:02.775 INFO:teuthology.orchestra.run.smithi046.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-02-22T02:30:02.779 DEBUG:teuthology.parallel:result is None 2024-02-22T02:30:02.780 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=dbe7fcec16eddee6239e4dd68c4d0203c7df6461 2024-02-22T02:30:02.921 DEBUG:teuthology.orchestra.run.smithi046:> dpkg-query -W -f '${Version}' ceph 2024-02-22T02:30:02.944 INFO:teuthology.orchestra.run.smithi046.stdout:19.0.0-1633-gdbe7fcec-1jammy 2024-02-22T02:30:02.944 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1633-gdbe7fcec-1jammy 2024-02-22T02:30:02.944 INFO:teuthology.task.install:The correct ceph version 19.0.0-1633-gdbe7fcec-1jammy is installed. 2024-02-22T02:30:02.947 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-22T02:30:02.947 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-02-22T02:30:02.947 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-22T02:30:02.999 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-22T02:30:02.999 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-02-22T02:30:03.000 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-02-22T02:30:03.092 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-22T02:30:03.147 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-22T02:30:03.147 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-02-22T02:30:03.147 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-22T02:30:03.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-22T02:30:03.263 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-22T02:30:03.263 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-02-22T02:30:03.263 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-02-22T02:30:03.320 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-22T02:30:03.375 INFO:teuthology.run_tasks:Running task workunit... 2024-02-22T02:30:03.390 INFO:tasks.workunit:Pulling workunits from ref dbe7fcec16eddee6239e4dd68c4d0203c7df6461 2024-02-22T02:30:03.390 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-22T02:30:03.390 INFO:tasks.workunit:timeout=3h 2024-02-22T02:30:03.390 INFO:tasks.workunit:cleanup=True 2024-02-22T02:30:03.391 DEBUG:teuthology.orchestra.run.smithi046:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-22T02:30:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-22T02:30:03.424 INFO:teuthology.orchestra.run.smithi046.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-02-22T02:30:03.424 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-02-22T02:30:03.471 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-02-22T02:30:03.471 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-02-22T02:30:03.520 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout dbe7fcec16eddee6239e4dd68c4d0203c7df6461 2024-02-22T02:30:03.573 INFO:tasks.workunit.client.0.smithi046.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-22T02:31:30.811 INFO:tasks.workunit.client.0.smithi046.stderr:Note: switching to 'dbe7fcec16eddee6239e4dd68c4d0203c7df6461'. 2024-02-22T02:31:30.811 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-02-22T02:31:30.811 INFO:tasks.workunit.client.0.smithi046.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-22T02:31:30.811 INFO:tasks.workunit.client.0.smithi046.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr:state without impacting any branches by switching back to a branch. 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr: git switch -c 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr:Or undo this operation with: 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr: git switch - 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-02-22T02:31:30.812 INFO:tasks.workunit.client.0.smithi046.stderr:HEAD is now at dbe7fcec16e mgr: integrate optracker in ceph-mgr 2024-02-22T02:31:30.814 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-22T02:31:30.865 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-02-22T02:31:30.865 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-22T02:31:30.911 INFO:tasks.workunit:Running workunits matching misc on client.0... 2024-02-22T02:31:30.911 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2024-02-22T02:31:30.912 DEBUG:teuthology.orchestra.run.smithi046:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbe7fcec16eddee6239e4dd68c4d0203c7df6461 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2024-02-22T02:31:30.977 INFO:tasks.workunit.client.0.smithi046.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-22T02:31:30.982 INFO:tasks.workunit.client.0.smithi046.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-22T02:31:30.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/mclock-config 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2024-02-22T02:31:30.983 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2024-02-22T02:31:30.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none ' 2024-02-22T02:31:30.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-02-22T02:31:30.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2024-02-22T02:31:30.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2024-02-22T02:31:30.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-22T02:31:30.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2024-02-22T02:31:30.986 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_profile_builtin_to_custom 2024-02-22T02:31:30.986 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_profile_custom_to_builtin 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_profile_disallow_builtin_params_modify 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_profile_disallow_builtin_params_override 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_recovery_limit_adjustment_mclock' 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:31:30.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:31:30.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:31:30.989 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:31:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:31:30.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:31:30.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:31:30.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:31:30.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:31:30.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:31:30.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:31:30.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:31:30.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:31:30.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:31:30.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:31:30.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:31:30.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:31:30.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:31:30.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:31:30.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:31:30.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:31:30.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:31:30.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-02-22T02:31:30.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:31:30.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:31:30.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:31:30.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20104 2024-02-22T02:31:30.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:31:31.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-22T02:31:31.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-22T02:31:31.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:31:31.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-02-22T02:31:31.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2024-02-22T02:31:31.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2024-02-22T02:31:31.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2024-02-22T02:31:31.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-02-22T02:31:31.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-02-22T02:31:31.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:31:31.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:31:31.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:31:31.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:31:31.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:31:31.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:31:31.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:31:31.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:31:31.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:31:31.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:31:31.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:31:31.004 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:31:31.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:31:31.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:31:31.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:31:31.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:31:31.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:31:31.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:31:31.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:31:31.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:31:31.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:31:31.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:31:31.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:31:31.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:31:31.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:31:31.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:31:31.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:31:31.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:31:31.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-02-22T02:31:31.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:31:31.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:31:31.012 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:31:31.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20104 2024-02-22T02:31:31.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:31:31.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:31:31.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:31:31.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-02-22T02:31:31.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2024-02-22T02:31:31.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-02-22T02:31:31.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:31:31.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:31:31.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:31:31.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-02-22T02:31:31.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-02-22T02:31:31.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:31:31.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:31:31.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:31:31.542 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:31:31.542 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:31:31.543 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:31:31.543 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:31:31.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-22T02:31:31.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:31:31.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:31:31.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:31:31.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:31:31.770 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:31:31.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:31:31.771 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:31:31.771 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:31:31.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:31:31.772 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:31:31.772 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:31:31.772 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:31:31.772 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-mon.a.asok 2024-02-22T02:31:31.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:31:31.772 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20104/ceph-mon.a.asok config get fsid 2024-02-22T02:31:31.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:31:31.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:31:31.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:31:31.889 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:31:31.890 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:31:31.890 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:31:31.891 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:31:31.891 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:31:31.891 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:31:31.891 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:31:31.891 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:31:31.891 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-mon.a.asok 2024-02-22T02:31:31.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:31:31.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20104/ceph-mon.a.asok config get mon_host 2024-02-22T02:31:32.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-02-22T02:31:32.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-02-22T02:31:32.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:31:32.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:31:32.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:31:32.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-02-22T02:31:32.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:31:32.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:31:32.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:31:32.946 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:31:32.946 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:31:32.947 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:31:32.947 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:31:32.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:31:32.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:31:32.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:31:33.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-02-22T02:31:33.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-02-22T02:31:33.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:31:33.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:31:33.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:31:33.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-02-22T02:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-02-22T02:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-02-22T02:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-02-22T02:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-02-22T02:31:33.029 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:31:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:31:33.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:31:33.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:31:33.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:31:33.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-02-22T02:31:33.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-02-22T02:31:33.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:31:33.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fd2b20fa-3aea-4d34-ae86-d163207f4e4f 2024-02-22T02:31:33.032 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 fd2b20fa-3aea-4d34-ae86-d163207f4e4f 2024-02-22T02:31:33.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fd2b20fa-3aea-4d34-ae86-d163207f4e4f' 2024-02-22T02:31:33.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:31:33.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAFstZlY0OeAhAAME9Pi4mHK+pEzf3pCpiHvw== 2024-02-22T02:31:33.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAFstZlY0OeAhAAME9Pi4mHK+pEzf3pCpiHvw=="}' 2024-02-22T02:31:33.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fd2b20fa-3aea-4d34-ae86-d163207f4e4f -i td/mclock-config/0/new.json 2024-02-22T02:31:33.366 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:31:33.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-02-22T02:31:33.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAFstZlY0OeAhAAME9Pi4mHK+pEzf3pCpiHvw== --osd-uuid fd2b20fa-3aea-4d34-ae86-d163207f4e4f 2024-02-22T02:31:33.467 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:31:33.463+0000 7f562727d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:31:33.469 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:31:33.463+0000 7f562727d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:31:33.471 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:31:33.467+0000 7f562727d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:31:33.471 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:31:33.467+0000 7f562727d7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:31:33.471 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:31:33.467+0000 7f562727d7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:31:33.471 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:31:33.467+0000 7f562727d7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:31:33.510 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:31:33.503+0000 7f562727d7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-02-22T02:31:40.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-02-22T02:31:40.420 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:31:40.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:31:40.423 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:31:40.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:31:40.663 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:31:40.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:31:40.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:31:40.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-02-22T02:31:40.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:31:40.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:31:40.696 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:31:40.691+0000 7fd0c2d8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:31:40.698 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:31:40.695+0000 7fd0c2d8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:31:40.700 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:31:40.695+0000 7fd0c2d8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:31:40.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:31:40.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:31:40.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:31:40.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:31:40.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:31:40.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:31:40.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:31:40.891 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:31:40.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:31:40.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:31:41.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:31:41.264 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:31:41.259+0000 7fd0c2d8f7c0 -1 Falling back to public interface 2024-02-22T02:31:42.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:31:42.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:31:42.174 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:31:42.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:31:42.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:31:42.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:31:42.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:31:43.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:31:43.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:31:43.486 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:31:43.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:31:43.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:31:43.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:31:43.682 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:31:43.679+0000 7fd0c2d8f7c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:31:43.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:31:44.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:31:44.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:31:44.917 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:31:44.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:31:44.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:31:44.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:31:45.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:31:46.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:31:46.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:31:46.564 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:31:46.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:31:46.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:31:46.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:31:46.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:31:47.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:31:47.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:31:47.996 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:31:47.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:31:47.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:31:47.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:31:48.560 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:31:49.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:31:49.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:31:49.562 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:31:49.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:31:49.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:31:49.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:31:49.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:31:50.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:31:50.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:31:50.985 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:31:50.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:31:50.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:31:50.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:31:51.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:31:52.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:31:52.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:31:52.556 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:31:52.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:31:52.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:31:52.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:31:53.005 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:31:54.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:31:54.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:31:54.007 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:31:54.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:31:54.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:31:54.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:31:54.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:31:55.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:31:55.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:31:55.701 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:31:55.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:31:55.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:31:55.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:31:56.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:31:57.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:31:57.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:31:57.147 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:31:57.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:31:57.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:31:57.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:31:58.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:31:59.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:31:59.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:31:59.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:31:59.256 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:31:59.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:31:59.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:31:59.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:00.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:00.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:00.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:32:00.829 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:32:00.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:00.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:01.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:02.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:02.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:02.426 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:32:02.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:32:02.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:02.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:03.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:04.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:04.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:04.827 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:32:04.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:32:04.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:04.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:05.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:06.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:06.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:06.831 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:32:06.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:32:06.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:06.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:07.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:08.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:08.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:08.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:32:08.642 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:32:08.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:08.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:09.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:10.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:10.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:10.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-22T02:32:10.872 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-02-22T02:32:10.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:10.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:11.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:12.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:12.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:12.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-22T02:32:12.404 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-02-22T02:32:12.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:12.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:12.819 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/802859455,v1:127.0.0.1:6801/802859455] [v2:127.0.0.1:6802/802859455,v1:127.0.0.1:6803/802859455] exists,up fd2b20fa-3aea-4d34-ae86-d163207f4e4f 2024-02-22T02:32:12.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:32:12.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:32:12.819 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:32:12.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-02-22T02:32:12.820 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2024-02-22T02:32:12.821 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-02-22T02:32:12.821 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:32:12.821 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:32:12.822 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:32:12.822 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:12.822 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:12.822 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:32:12.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-02-22T02:32:12.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_max_backfills 2024-02-22T02:32:12.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2024-02-22T02:32:12.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2024-02-22T02:32:12.945 INFO:tasks.workunit.client.0.smithi046.stdout:osd_max_backfills: 1 2024-02-22T02:32:12.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-02-22T02:32:13.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-02-22T02:32:15.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-02-22T02:32:15.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2024-02-22T02:32:15.645 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-02-22T02:32:15.645 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:32:15.645 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:32:15.646 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:32:15.646 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:15.646 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:15.646 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:32:15.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-02-22T02:32:15.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_max_backfills 2024-02-22T02:32:15.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2024-02-22T02:32:15.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-02-22T02:32:15.777 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2024-02-22T02:32:15.778 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-02-22T02:32:15.779 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-02-22T02:32:15.779 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:32:15.779 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:32:15.779 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:32:15.779 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:15.779 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:15.779 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:32:15.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-02-22T02:32:15.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok dump_recovery_reservations 2024-02-22T02:32:15.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-02-22T02:32:15.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-02-22T02:32:15.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-02-22T02:32:15.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2024-02-22T02:32:15.902 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-02-22T02:32:15.902 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:32:15.902 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:32:15.903 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:32:15.903 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:15.903 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:15.903 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:32:15.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-02-22T02:32:15.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok dump_recovery_reservations 2024-02-22T02:32:16.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-02-22T02:32:16.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-02-22T02:32:16.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-02-22T02:32:16.528 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-02-22T02:32:17.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-02-22T02:32:19.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-02-22T02:32:19.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2024-02-22T02:32:19.060 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-02-22T02:32:19.060 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:32:19.060 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:32:19.061 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:32:19.061 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:19.061 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:19.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:32:19.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-02-22T02:32:19.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_max_backfills 2024-02-22T02:32:19.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-02-22T02:32:19.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-02-22T02:32:19.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-02-22T02:32:19.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2024-02-22T02:32:19.190 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-02-22T02:32:19.190 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:32:19.190 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:32:19.190 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:32:19.190 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:19.190 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:19.190 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:32:19.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-02-22T02:32:19.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok dump_recovery_reservations 2024-02-22T02:32:19.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-02-22T02:32:19.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-02-22T02:32:19.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-02-22T02:32:19.312 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-02-22T02:32:19.312 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:32:19.312 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:32:19.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2024-02-22T02:32:19.312 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:32:19.312 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:19.312 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:19.313 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:32:19.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-02-22T02:32:19.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok dump_recovery_reservations 2024-02-22T02:32:19.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-02-22T02:32:19.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-02-22T02:32:19.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2024-02-22T02:32:19.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:32:19.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:32:19.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:32:19.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:32:19.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:32:19.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:32:19.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2024-02-22T02:32:20.220 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 is already down. 2024-02-22T02:32:20.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2024-02-22T02:32:20.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-02-22T02:32:20.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:32:20.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:32:20.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:32:20.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:20.238 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:32:20.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:32:20.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:20.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-02-22T02:32:20.754 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6800/802859455,v1:127.0.0.1:6801/802859455] [v2:127.0.0.1:6802/802859455,v1:127.0.0.1:6803/802859455] exists fd2b20fa-3aea-4d34-ae86-d163207f4e4f 2024-02-22T02:32:20.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:32:20.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:32:20.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:32:20.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2024-02-22T02:32:20.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/mclock-config 2024-02-22T02:32:20.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-22T02:32:20.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-22T02:32:20.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-22T02:32:20.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/mclock-config/0 2024-02-22T02:32:20.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-02-22T02:32:20.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:32:20.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:32:20.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:32:20.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-02-22T02:32:20.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-02-22T02:32:20.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-22T02:32:20.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-22T02:32:20.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-02-22T02:32:20.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-22T02:32:20.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:32:20.757 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:32:20.759 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:32:20.759 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:20.759 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:20.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:32:20.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:32:20.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:32:20.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-02-22T02:32:20.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-02-22T02:32:20.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:32:20.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:32:20.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:32:20.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:32:20.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-22T02:32:20.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2024-02-22T02:32:20.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/mclock-config/0 2024-02-22T02:32:20.760 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:32:20.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-22T02:32:20.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2024-02-22T02:32:20.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/mclock-config/0/whoami 2024-02-22T02:32:20.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-22T02:32:20.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-22T02:32:20.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-22T02:32:20.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-22T02:32:20.790 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:20.788+0000 7f5aa959b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:32:20.827 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:20.824+0000 7f5aa959b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:32:20.828 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:20.824+0000 7f5aa959b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:32:21.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-22T02:32:21.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:32:21.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:32:21.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:32:21.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:32:21.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:21.232 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:32:21.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:32:21.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:21.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:21.375 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:21.372+0000 7f5aa959b7c0 -1 Falling back to public interface 2024-02-22T02:32:21.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:22.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:22.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:22.696 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:32:22.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:32:22.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:22.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:23.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:24.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:24.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:24.123 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:32:24.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:32:24.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:24.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:24.450 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:24.448+0000 7f5aa959b7c0 -1 osd.0 6 log_to_monitors true 2024-02-22T02:32:24.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:25.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:25.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:25.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:32:25.551 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:32:25.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:25.551 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:26.185 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6800/2450772133,v1:127.0.0.1:6801/2450772133] [v2:127.0.0.1:6802/2450772133,v1:127.0.0.1:6803/2450772133] exists,up fd2b20fa-3aea-4d34-ae86-d163207f4e4f 2024-02-22T02:32:26.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:32:26.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:32:26.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:32:26.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2024-02-22T02:32:26.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-02-22T02:32:26.187 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-02-22T02:32:26.187 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:32:26.187 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:32:26.187 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:32:26.187 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:26.188 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:26.188 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:32:26.188 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-02-22T02:32:26.188 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_max_backfills 2024-02-22T02:32:26.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-02-22T02:32:26.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-02-22T02:32:26.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-02-22T02:32:26.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2024-02-22T02:32:26.311 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-02-22T02:32:26.311 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:32:26.311 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:32:26.312 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:32:26.312 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:26.312 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:26.312 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:32:26.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-02-22T02:32:26.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok dump_recovery_reservations 2024-02-22T02:32:26.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-02-22T02:32:26.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-02-22T02:32:26.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-02-22T02:32:26.434 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-02-22T02:32:26.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2024-02-22T02:32:26.434 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:32:26.434 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:32:26.434 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:32:26.434 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:26.434 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:26.434 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:32:26.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-02-22T02:32:26.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok dump_recovery_reservations 2024-02-22T02:32:26.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-02-22T02:32:26.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-02-22T02:32:26.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2024-02-22T02:32:26.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:32:26.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:32:26.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:32:26.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:32:26.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:32:26.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:32:26.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:32:26.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:32:26.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:32:26.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:32:26.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:32:26.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:32:26.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:32:26.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:32:26.871 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:32:26.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:32:26.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:32:26.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:32:26.873 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:32:26.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:32:26.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:32:26.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:32:26.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:32:26.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:26.885 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:26.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:32:26.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:32:26.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:32:26.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-02-22T02:32:26.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:32:26.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:32:26.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:32:26.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:32:26.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:32:26.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:32:26.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:32:26.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:32:26.890 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:32:26.891 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:32:26.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:32:26.892 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:32:26.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:32:26.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:32:26.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:32:26.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:32:26.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:32:26.896 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:32:26.896 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:32:26.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:32:26.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:32:26.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:32:26.900 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:32:26.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:26.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:26.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:32:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:32:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:32:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-02-22T02:32:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-02-22T02:32:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-02-22T02:32:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-02-22T02:32:26.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:32:26.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:32:26.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:32:26.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:32:26.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:32:26.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:32:26.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:32:26.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:32:26.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:32:26.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:32:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:32:26.907 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:32:26.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:32:26.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:32:26.909 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:32:26.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:32:26.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:32:26.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:32:26.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:32:26.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:32:26.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:32:26.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:32:26.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:32:26.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:26.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:26.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:32:26.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:32:26.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:32:26.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-02-22T02:32:26.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:32:26.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:26.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:26.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20104 2024-02-22T02:32:26.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:32:26.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:32:26.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:32:26.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-02-22T02:32:26.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2024-02-22T02:32:26.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2024-02-22T02:32:26.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2024-02-22T02:32:26.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-02-22T02:32:26.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:32:26.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:32:26.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:32:26.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-02-22T02:32:26.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-02-22T02:32:27.289 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:32:27.289 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:32:27.289 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:32:27.289 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:32:27.289 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:27.289 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:27.289 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:32:27.290 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-22T02:32:27.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:32:27.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:32:27.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:32:27.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:32:27.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:32:27.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:32:27.511 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:32:27.511 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:32:27.511 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:32:27.511 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:32:27.511 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:27.512 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:27.512 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-mon.a.asok 2024-02-22T02:32:27.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:32:27.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20104/ceph-mon.a.asok config get fsid 2024-02-22T02:32:27.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:32:27.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:32:27.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:32:27.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:32:27.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:32:27.631 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:32:27.631 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:32:27.631 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:32:27.631 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:32:27.631 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:27.631 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:27.632 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-mon.a.asok 2024-02-22T02:32:27.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:32:27.632 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20104/ceph-mon.a.asok config get mon_host 2024-02-22T02:32:27.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2024-02-22T02:32:27.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-02-22T02:32:27.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:32:27.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:32:27.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:32:27.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-02-22T02:32:27.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:32:28.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:32:28.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:32:28.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:32:28.665 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:32:28.665 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:28.665 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:28.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:32:28.666 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:32:28.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:32:28.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:32:28.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:32:28.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:32:28.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:32:28.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:32:28.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:32:28.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-02-22T02:32:28.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-02-22T02:32:28.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:32:28.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=701974ed-19bf-4031-b1c5-82a6cdd6d42f 2024-02-22T02:32:28.709 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 701974ed-19bf-4031-b1c5-82a6cdd6d42f 2024-02-22T02:32:28.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 701974ed-19bf-4031-b1c5-82a6cdd6d42f' 2024-02-22T02:32:28.710 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:32:28.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8stZlRIAfKxAA8cF4FZbLWBVNYXJIgHmCkg== 2024-02-22T02:32:28.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8stZlRIAfKxAA8cF4FZbLWBVNYXJIgHmCkg=="}' 2024-02-22T02:32:28.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 701974ed-19bf-4031-b1c5-82a6cdd6d42f -i td/mclock-config/0/new.json 2024-02-22T02:32:29.045 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:32:29.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-02-22T02:32:29.061 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQA8stZlRIAfKxAA8cF4FZbLWBVNYXJIgHmCkg== --osd-uuid 701974ed-19bf-4031-b1c5-82a6cdd6d42f 2024-02-22T02:32:29.089 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:29.088+0000 7fa922ffa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:32:29.122 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:29.120+0000 7fa922ffa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:32:29.123 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:29.120+0000 7fa922ffa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:32:29.123 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:29.120+0000 7fa922ffa7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:32:29.123 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:29.120+0000 7fa922ffa7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:32:29.123 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:29.120+0000 7fa922ffa7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:32:29.162 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:29.160+0000 7fa922ffa7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-02-22T02:32:35.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-02-22T02:32:35.687 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:32:35.688 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:32:35.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:32:35.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:32:36.246 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:32:36.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:32:36.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:32:36.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-02-22T02:32:36.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:32:36.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:32:36.279 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:36.276+0000 7fee903197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:32:36.306 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:36.304+0000 7fee903197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:32:36.308 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:36.304+0000 7fee903197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:32:36.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:32:36.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:32:36.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:32:36.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:32:36.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:32:36.676 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:36.676 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:32:36.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:32:36.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:36.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:36.868 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:36.864+0000 7fee903197c0 -1 Falling back to public interface 2024-02-22T02:32:37.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:38.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:38.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:38.129 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:32:38.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:32:38.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:38.129 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:38.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:39.501 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:32:39.500+0000 7fee903197c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:32:39.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:39.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:39.542 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:32:39.542 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:32:39.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:39.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:40.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:41.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:41.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:41.046 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:32:41.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:32:41.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:41.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:41.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:42.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:42.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:42.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:32:42.524 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:32:42.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:42.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:42.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:43.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:43.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:43.974 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:32:43.974 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:32:43.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:43.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:44.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:45.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:45.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:45.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:32:45.515 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:32:45.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:45.516 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:45.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:46.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:46.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:46.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:32:46.962 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:32:46.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:46.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:47.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:48.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:48.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:48.572 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:32:48.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:32:48.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:48.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:49.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:50.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:50.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:50.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:32:50.016 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:32:50.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:50.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:50.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:51.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:51.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:51.545 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:32:51.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:32:51.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:51.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:52.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:53.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:53.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:53.028 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:32:53.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:32:53.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:53.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:53.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:54.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:54.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:54.863 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:32:54.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:32:54.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:54.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:55.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:56.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:56.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:56.749 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:32:56.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:32:56.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:56.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:57.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:32:58.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:32:58.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:32:58.534 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:32:58.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:32:58.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:32:58.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:32:59.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:00.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:00.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:00.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:33:00.480 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:33:00.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:00.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:01.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:02.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:02.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:02.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:33:02.525 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:33:02.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:02.525 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:03.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:04.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:04.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:04.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:33:04.464 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:33:04.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:04.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:04.920 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/280949093,v1:127.0.0.1:6803/280949093] [v2:127.0.0.1:6804/280949093,v1:127.0.0.1:6805/280949093] exists,up 701974ed-19bf-4031-b1c5-82a6cdd6d42f 2024-02-22T02:33:04.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:33:04.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:33:04.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:33:04.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2024-02-22T02:33:05.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2024-02-22T02:33:05.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2024-02-22T02:33:05.356 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-02-22T02:33:05.356 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-02-22T02:33:05.356 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:33:05.356 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:33:05.356 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:05.357 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:05.357 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:05.357 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:33:05.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-02-22T02:33:05.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_profile 2024-02-22T02:33:05.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2024-02-22T02:33:05.478 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2024-02-22T02:33:05.478 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2024-02-22T02:33:05.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2024-02-22T02:33:05.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2024-02-22T02:33:05.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2024-02-22T02:33:05.626 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:33:05.626 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "" 2024-02-22T02:33:05.626 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:33:05.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-02-22T02:33:05.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-02-22T02:33:05.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:33:05.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:33:05.642 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:05.643 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:05.643 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:05.643 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:33:05.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-02-22T02:33:05.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_profile 2024-02-22T02:33:05.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2024-02-22T02:33:05.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2024-02-22T02:33:05.759 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2024-02-22T02:33:05.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2024-02-22T02:33:05.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2024-02-22T02:33:05.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-02-22T02:33:05.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2024-02-22T02:33:05.761 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-02-22T02:33:05.761 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:33:05.761 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:33:05.761 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:05.761 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:05.761 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:05.762 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:33:05.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-02-22T02:33:05.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-02-22T02:33:05.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2024-02-22T02:33:05.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2024-02-22T02:33:05.880 INFO:tasks.workunit.client.0.smithi046.stdout:client_res = 0.300000 2024-02-22T02:33:05.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2024-02-22T02:33:05.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2024-02-22T02:33:05.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2024-02-22T02:33:05.882 INFO:tasks.workunit.client.0.smithi046.stdout:client_res_new = .400000 2024-02-22T02:33:05.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2024-02-22T02:33:05.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-02-22T02:33:06.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-02-22T02:33:06.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2024-02-22T02:33:06.810 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2024-02-22T02:33:06.811 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-02-22T02:33:06.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2024-02-22T02:33:06.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-02-22T02:33:06.814 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-02-22T02:33:06.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2024-02-22T02:33:06.814 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:33:06.814 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:33:06.814 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:06.814 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:06.814 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:06.814 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:33:06.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-02-22T02:33:06.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-02-22T02:33:06.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2024-02-22T02:33:06.929 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-02-22T02:33:06.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2024-02-22T02:33:06.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2024-02-22T02:33:06.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2024-02-22T02:33:06.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:33:06.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:33:06.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:33:06.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:33:06.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:33:06.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:33:06.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:33:06.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:33:07.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:33:07.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:33:07.244 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:33:07.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:33:07.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:33:07.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:33:07.246 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:33:07.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:07.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:33:07.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:33:07.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:07.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:33:07.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:33:07.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:33:07.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:33:07.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:07.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:07.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:33:07.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:33:07.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:33:07.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-02-22T02:33:07.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:33:07.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:33:07.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:33:07.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:33:07.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:33:07.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:33:07.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:33:07.263 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:33:07.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:33:07.265 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:33:07.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:33:07.267 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:33:07.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:33:07.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:33:07.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:33:07.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:07.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:33:07.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:33:07.270 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:07.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:33:07.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:33:07.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:33:07.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:33:07.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:07.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:07.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:33:07.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:33:07.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:33:07.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-02-22T02:33:07.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-02-22T02:33:07.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-02-22T02:33:07.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-02-22T02:33:07.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:33:07.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:33:07.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:33:07.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:33:07.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:33:07.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:33:07.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:33:07.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:33:07.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:33:07.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:33:07.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:33:07.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:33:07.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:33:07.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:33:07.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:33:07.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:07.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:33:07.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:33:07.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:07.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:33:07.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:33:07.289 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:33:07.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:33:07.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:07.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:07.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:33:07.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:33:07.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:33:07.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-02-22T02:33:07.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:33:07.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:07.294 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:07.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20104 2024-02-22T02:33:07.296 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:33:07.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:33:07.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:33:07.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-02-22T02:33:07.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2024-02-22T02:33:07.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2024-02-22T02:33:07.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2024-02-22T02:33:07.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-02-22T02:33:07.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-02-22T02:33:07.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:33:07.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:33:07.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:33:07.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:33:07.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:33:07.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:33:07.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:33:07.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:33:07.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:33:07.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:33:07.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:33:07.302 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:33:07.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:33:07.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:33:07.304 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:33:07.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:07.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:33:07.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:33:07.306 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:07.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:33:07.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:33:07.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:33:07.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:33:07.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:07.310 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:07.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:33:07.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:33:07.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:33:07.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-02-22T02:33:07.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:33:07.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:07.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:07.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20104 2024-02-22T02:33:07.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:33:07.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:33:07.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:33:07.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-02-22T02:33:07.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2024-02-22T02:33:07.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-02-22T02:33:07.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:33:07.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:33:07.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:33:07.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-02-22T02:33:07.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-02-22T02:33:07.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:33:07.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:33:07.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:33:07.715 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:33:07.715 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:07.715 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:07.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:33:07.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-22T02:33:07.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:33:07.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:33:07.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:33:07.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:33:07.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:33:07.952 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:33:07.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:33:07.952 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:33:07.952 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:33:07.953 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:07.953 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:07.953 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:07.953 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-mon.a.asok 2024-02-22T02:33:07.953 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:33:07.954 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20104/ceph-mon.a.asok config get fsid 2024-02-22T02:33:08.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:33:08.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:33:08.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:33:08.073 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:33:08.074 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:33:08.074 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:33:08.074 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:33:08.074 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:33:08.075 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:08.075 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:08.075 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:08.075 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-mon.a.asok 2024-02-22T02:33:08.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:33:08.076 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20104/ceph-mon.a.asok config get mon_host 2024-02-22T02:33:08.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2024-02-22T02:33:08.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-02-22T02:33:08.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:33:08.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:33:08.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:33:08.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-02-22T02:33:08.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:33:09.113 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:33:09.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:33:09.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:33:09.114 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:33:09.114 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:09.114 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:09.114 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:33:09.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:33:09.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:33:09.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-02-22T02:33:09.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-02-22T02:33:09.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:33:09.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:33:09.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:33:09.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-02-22T02:33:09.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-02-22T02:33:09.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:09.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:33:09.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:33:09.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:33:09.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:33:09.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:33:09.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-02-22T02:33:09.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-02-22T02:33:09.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:33:09.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:33:09.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:33:09.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:33:09.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:33:09.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-02-22T02:33:09.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-02-22T02:33:09.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:33:09.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3d231dae-e9e0-41c4-9e80-98ed226229b2 2024-02-22T02:33:09.178 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 3d231dae-e9e0-41c4-9e80-98ed226229b2 2024-02-22T02:33:09.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3d231dae-e9e0-41c4-9e80-98ed226229b2' 2024-02-22T02:33:09.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:33:09.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlstZlh3iTCxAAMSW0ugcnxtr5+Iup7D4ydg== 2024-02-22T02:33:09.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlstZlh3iTCxAAMSW0ugcnxtr5+Iup7D4ydg=="}' 2024-02-22T02:33:09.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3d231dae-e9e0-41c4-9e80-98ed226229b2 -i td/mclock-config/0/new.json 2024-02-22T02:33:09.488 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:33:09.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-02-22T02:33:09.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBlstZlh3iTCxAAMSW0ugcnxtr5+Iup7D4ydg== --osd-uuid 3d231dae-e9e0-41c4-9e80-98ed226229b2 2024-02-22T02:33:09.537 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:09.536+0000 7f67265eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:33:09.539 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:09.536+0000 7f67265eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:33:09.540 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:09.536+0000 7f67265eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:33:09.541 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:09.540+0000 7f67265eb7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:33:09.541 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:09.540+0000 7f67265eb7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:33:09.541 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:09.540+0000 7f67265eb7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:33:09.612 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:09.608+0000 7f67265eb7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-02-22T02:33:15.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-02-22T02:33:15.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:33:16.000 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:33:16.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:33:16.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:33:16.595 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:33:16.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:33:16.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:33:16.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:33:16.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:33:16.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-02-22T02:33:16.627 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:16.624+0000 7f7e785677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:33:16.629 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:16.628+0000 7f7e785677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:33:16.631 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:16.628+0000 7f7e785677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:33:17.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:33:17.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:33:17.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:33:17.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:33:17.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:33:17.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:17.044 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:33:17.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:33:17.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:17.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:17.194 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:17.192+0000 7f7e785677c0 -1 Falling back to public interface 2024-02-22T02:33:17.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:18.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:18.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:18.463 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:33:18.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:33:18.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:18.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:18.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:19.851 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:19.848+0000 7f7e785677c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:33:19.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:19.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:19.929 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:33:19.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:33:19.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:19.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:20.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:21.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:21.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:21.568 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:33:21.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:33:21.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:21.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:22.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:23.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:23.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:23.029 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:33:23.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:33:23.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:23.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:23.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:24.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:24.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:24.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:33:24.490 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:33:24.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:24.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:24.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:25.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:25.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:25.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:33:25.997 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:33:25.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:25.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:26.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:27.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:27.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:27.446 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:33:27.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:33:27.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:27.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:27.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:28.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:28.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:28.968 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:33:28.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:33:28.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:28.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:29.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:30.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:30.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:30.410 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:33:30.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:33:30.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:30.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:30.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:31.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:31.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:31.829 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:33:31.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:33:31.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:31.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:32.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:33.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:33.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:33.258 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:33:33.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:33:33.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:33.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:33.987 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:34.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:34.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:34.990 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:33:34.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:33:34.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:34.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:36.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:37.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:37.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:37.085 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:33:37.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:33:37.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:37.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:37.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:38.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:38.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:38.775 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:33:38.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:33:38.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:38.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:39.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:40.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:40.530 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:40.531 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:33:40.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:33:40.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:40.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:41.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:42.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:42.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:42.069 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:33:42.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:33:42.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:42.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:43.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:33:44.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:33:44.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:33:44.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:33:44.478 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:33:44.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:33:44.479 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:33:44.897 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/289093758,v1:127.0.0.1:6803/289093758] [v2:127.0.0.1:6804/289093758,v1:127.0.0.1:6805/289093758] exists,up 3d231dae-e9e0-41c4-9e80-98ed226229b2 2024-02-22T02:33:44.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:33:44.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:33:44.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:33:44.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2024-02-22T02:33:44.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2024-02-22T02:33:45.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2024-02-22T02:33:45.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2024-02-22T02:33:45.372 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-02-22T02:33:45.373 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-02-22T02:33:45.373 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:33:45.373 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:33:45.374 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:45.374 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:45.374 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:45.374 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:33:45.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-02-22T02:33:45.375 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_profile 2024-02-22T02:33:45.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2024-02-22T02:33:45.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2024-02-22T02:33:45.501 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2024-02-22T02:33:45.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2024-02-22T02:33:45.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-02-22T02:33:45.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2024-02-22T02:33:45.644 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:33:45.644 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "" 2024-02-22T02:33:45.644 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:33:45.660 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-02-22T02:33:45.660 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-02-22T02:33:45.660 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:33:45.660 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:33:45.660 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:45.660 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:45.660 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:45.660 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:33:45.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-02-22T02:33:45.661 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_profile 2024-02-22T02:33:45.778 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2024-02-22T02:33:45.779 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2024-02-22T02:33:45.779 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2024-02-22T02:33:45.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2024-02-22T02:33:45.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2024-02-22T02:33:45.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2024-02-22T02:33:45.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2024-02-22T02:33:45.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-02-22T02:33:45.781 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-02-22T02:33:45.781 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:33:45.781 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:33:45.782 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:45.782 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:45.782 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:45.782 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:33:45.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-02-22T02:33:45.782 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-02-22T02:33:45.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2024-02-22T02:33:45.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2024-02-22T02:33:45.910 INFO:tasks.workunit.client.0.smithi046.stdout:Original client_res for osd.0 = 0.300000 2024-02-22T02:33:45.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2024-02-22T02:33:45.911 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2024-02-22T02:33:45.913 INFO:tasks.workunit.client.0.smithi046.stdout:client_res_new = .400000 2024-02-22T02:33:45.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2024-02-22T02:33:45.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2024-02-22T02:33:45.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2024-02-22T02:33:46.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-02-22T02:33:46.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2024-02-22T02:33:46.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-02-22T02:33:46.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2024-02-22T02:33:46.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2024-02-22T02:33:46.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-02-22T02:33:46.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2024-02-22T02:33:46.829 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-02-22T02:33:46.829 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:33:46.829 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:33:46.830 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:46.830 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:46.830 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:46.830 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:33:46.830 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-02-22T02:33:46.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-02-22T02:33:46.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2024-02-22T02:33:46.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-02-22T02:33:46.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2024-02-22T02:33:46.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2024-02-22T02:33:46.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2024-02-22T02:33:47.101 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:33:47.101 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "" 2024-02-22T02:33:47.101 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:33:47.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-02-22T02:33:47.118 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-02-22T02:33:47.118 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:33:47.118 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:33:47.118 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:47.118 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:47.118 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:47.118 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:33:47.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-02-22T02:33:47.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_profile 2024-02-22T02:33:47.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2024-02-22T02:33:47.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2024-02-22T02:33:47.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-02-22T02:33:47.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-02-22T02:33:47.650 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2024-02-22T02:33:47.651 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-02-22T02:33:47.651 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2024-02-22T02:33:47.652 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2024-02-22T02:33:47.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-02-22T02:33:47.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2024-02-22T02:33:47.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-02-22T02:33:47.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:33:47.655 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:33:47.655 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:47.656 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:47.656 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:47.656 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:33:47.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-02-22T02:33:47.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-02-22T02:33:47.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2024-02-22T02:33:47.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-02-22T02:33:47.771 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2024-02-22T02:33:47.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2024-02-22T02:33:47.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2024-02-22T02:33:48.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2024-02-22T02:33:53.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-02-22T02:33:53.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2024-02-22T02:33:53.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2024-02-22T02:33:53.755 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2024-02-22T02:33:53.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2024-02-22T02:33:53.759 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-02-22T02:33:53.759 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-02-22T02:33:53.759 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:33:53.760 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:33:53.760 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2024-02-22T02:33:53.760 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:53.761 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:53.761 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:53.761 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:33:53.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-02-22T02:33:53.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-02-22T02:33:53.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2024-02-22T02:33:53.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2024-02-22T02:33:53.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2024-02-22T02:33:53.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2024-02-22T02:33:53.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2024-02-22T02:33:53.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:33:53.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:33:53.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:33:53.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:33:53.883 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:33:53.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:33:53.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:33:53.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:33:54.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:33:54.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:33:54.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:33:54.194 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:33:54.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:33:54.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:33:54.197 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:33:54.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:54.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:33:54.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:33:54.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:54.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:33:54.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:33:54.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:33:54.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:33:54.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:54.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:54.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:33:54.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:33:54.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:33:54.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-02-22T02:33:54.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:33:54.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:33:54.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:33:54.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:33:54.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:33:54.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:33:54.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:33:54.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:33:54.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:33:54.215 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:33:54.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:33:54.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:33:54.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:33:54.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:33:54.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:33:54.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:54.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:33:54.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:33:54.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:54.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:33:54.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:33:54.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:33:54.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:33:54.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:54.224 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:54.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:33:54.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:33:54.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:33:54.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-02-22T02:33:54.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-02-22T02:33:54.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-02-22T02:33:54.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-02-22T02:33:54.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:33:54.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:33:54.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:33:54.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:33:54.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:33:54.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:33:54.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:33:54.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:33:54.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:33:54.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:33:54.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:33:54.232 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:33:54.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:33:54.233 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:33:54.234 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:33:54.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:54.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:33:54.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:33:54.236 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:54.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:33:54.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:33:54.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:33:54.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:33:54.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:54.240 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:54.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:33:54.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:33:54.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:33:54.241 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-02-22T02:33:54.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:33:54.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:54.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:54.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20104 2024-02-22T02:33:54.245 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:33:54.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:33:54.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:33:54.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:33:54.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:33:54.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:33:54.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:33:54.251 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:33:54.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:33:54.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:33:54.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:33:54.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:54.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:33:54.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:33:54.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:33:54.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:33:54.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:33:54.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:33:54.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:33:54.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:54.259 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:54.259 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:33:54.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:33:54.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:33:54.260 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-02-22T02:33:54.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:33:54.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:54.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:54.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20104 2024-02-22T02:33:54.264 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:33:54.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:33:54.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:33:54.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-02-22T02:33:54.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2024-02-22T02:33:54.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-02-22T02:33:54.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:33:54.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:33:54.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:33:54.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-02-22T02:33:54.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-02-22T02:33:54.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:33:54.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:33:54.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:33:54.627 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:33:54.627 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:54.627 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:54.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:33:54.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-22T02:33:54.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:33:54.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:33:54.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:33:54.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:33:54.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:33:54.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:33:54.850 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:33:54.850 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:33:54.850 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:33:54.851 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:54.851 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:54.851 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:54.852 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-mon.a.asok 2024-02-22T02:33:54.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:33:54.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20104/ceph-mon.a.asok config get fsid 2024-02-22T02:33:54.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:33:54.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:33:54.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:33:54.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:33:54.971 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:33:54.971 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:33:54.971 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:33:54.971 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:33:54.972 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:33:54.972 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:54.972 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:54.972 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-mon.a.asok 2024-02-22T02:33:54.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:33:54.972 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20104/ceph-mon.a.asok config get mon_host 2024-02-22T02:33:55.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2024-02-22T02:33:55.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-02-22T02:33:55.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:33:55.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:33:55.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:33:55.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-02-22T02:33:55.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:33:55.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:33:55.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:33:55.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:33:55.994 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:33:55.994 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:55.995 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:55.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:33:55.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:33:55.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:33:56.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-02-22T02:33:56.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-02-22T02:33:56.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:33:56.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:33:56.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:33:56.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-02-22T02:33:56.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-02-22T02:33:56.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:33:56.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:33:56.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:33:56.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:33:56.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:33:56.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:33:56.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-02-22T02:33:56.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-02-22T02:33:56.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:33:56.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:33:56.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:33:56.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:33:56.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:33:56.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-02-22T02:33:56.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-02-22T02:33:56.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:33:56.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b8df8c14-43c2-42bf-a512-d08566bf8c7a 2024-02-22T02:33:56.039 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 b8df8c14-43c2-42bf-a512-d08566bf8c7a 2024-02-22T02:33:56.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b8df8c14-43c2-42bf-a512-d08566bf8c7a' 2024-02-22T02:33:56.040 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:33:56.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCUstZlPU05AxAA/exKh5hkKyQW5TgdfsKHRA== 2024-02-22T02:33:56.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCUstZlPU05AxAA/exKh5hkKyQW5TgdfsKHRA=="}' 2024-02-22T02:33:56.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b8df8c14-43c2-42bf-a512-d08566bf8c7a -i td/mclock-config/0/new.json 2024-02-22T02:33:56.376 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:33:56.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-02-22T02:33:56.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCUstZlPU05AxAA/exKh5hkKyQW5TgdfsKHRA== --osd-uuid b8df8c14-43c2-42bf-a512-d08566bf8c7a 2024-02-22T02:33:56.423 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:56.419+0000 7fcf6a5f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:33:56.444 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:56.439+0000 7fcf6a5f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:33:56.445 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:56.443+0000 7fcf6a5f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:33:56.445 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:56.443+0000 7fcf6a5f37c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:33:56.445 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:56.443+0000 7fcf6a5f37c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:33:56.445 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:56.443+0000 7fcf6a5f37c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:33:56.483 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:33:56.479+0000 7fcf6a5f37c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-02-22T02:34:03.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-02-22T02:34:03.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:34:03.015 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:34:03.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:34:03.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:34:03.575 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:34:03.576 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:34:03.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-02-22T02:34:03.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:34:03.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:34:03.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:34:03.608 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:03.603+0000 7ff7e2f3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:34:03.643 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:03.639+0000 7ff7e2f3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:34:03.644 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:03.639+0000 7ff7e2f3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:34:04.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:34:04.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:34:04.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:34:04.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:34:04.019 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:34:04.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:34:04.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:04.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:34:04.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:04.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:04.230 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:04.227+0000 7ff7e2f3a7c0 -1 Falling back to public interface 2024-02-22T02:34:04.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:05.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:05.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:05.461 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:34:05.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:34:05.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:05.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:05.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:06.808 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:06.803+0000 7ff7e2f3a7c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:34:06.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:06.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:06.880 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:34:06.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:34:06.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:06.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:07.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:08.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:08.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:08.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:34:08.462 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:34:08.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:08.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:08.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:09.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:09.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:09.944 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:34:09.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:34:09.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:09.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:10.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:11.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:11.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:11.406 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:34:11.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:34:11.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:11.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:11.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:12.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:12.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:12.944 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:34:12.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:34:12.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:12.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:13.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:14.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:14.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:14.434 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:34:14.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:34:14.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:14.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:14.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:15.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:15.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:15.947 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:34:15.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:34:15.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:15.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:16.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:17.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:17.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:17.371 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:34:17.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:34:17.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:17.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:17.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:18.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:18.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:18.873 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:34:18.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:34:18.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:18.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:19.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:20.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:20.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:20.313 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:34:20.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:34:20.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:20.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:21.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:22.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:22.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:22.147 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:34:22.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:34:22.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:22.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:22.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:24.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:24.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:24.002 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:34:24.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:34:24.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:24.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:24.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:25.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:25.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:25.764 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:34:25.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:34:25.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:25.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:26.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:27.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:27.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:27.599 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:34:27.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:34:27.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:27.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:28.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:29.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:29.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:29.126 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:34:29.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:34:29.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:29.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:30.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:31.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:31.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:31.286 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:34:31.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:34:31.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:31.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:31.710 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2291486855,v1:127.0.0.1:6803/2291486855] [v2:127.0.0.1:6804/2291486855,v1:127.0.0.1:6805/2291486855] exists,up b8df8c14-43c2-42bf-a512-d08566bf8c7a 2024-02-22T02:34:31.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:34:31.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:34:31.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:34:31.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2024-02-22T02:34:32.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2024-02-22T02:34:32.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2024-02-22T02:34:32.159 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2024-02-22T02:34:32.159 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-02-22T02:34:32.159 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:34:32.159 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:34:32.160 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:34:32.160 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:32.160 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:32.160 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:34:32.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-02-22T02:34:32.160 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_profile 2024-02-22T02:34:32.277 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2024-02-22T02:34:32.278 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2024-02-22T02:34:32.278 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2024-02-22T02:34:32.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2024-02-22T02:34:32.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2024-02-22T02:34:32.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-02-22T02:34:32.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2024-02-22T02:34:32.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2024-02-22T02:34:32.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2024-02-22T02:34:32.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-02-22T02:34:32.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-02-22T02:34:32.280 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-02-22T02:34:32.280 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-02-22T02:34:32.280 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:34:32.280 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:34:32.281 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:34:32.281 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:32.281 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:32.281 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:34:32.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-02-22T02:34:32.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-02-22T02:34:32.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2024-02-22T02:34:32.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-02-22T02:34:32.400 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2024-02-22T02:34:32.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2024-02-22T02:34:32.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2024-02-22T02:34:33.012 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-02-22T02:34:33.013 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-02-22T02:34:33.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-02-22T02:34:33.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-02-22T02:34:33.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-02-22T02:34:35.047 INFO:tasks.workunit.client.0.smithi046.stdout:Check configuration values - Attempt#: 0 2024-02-22T02:34:35.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-02-22T02:34:35.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-02-22T02:34:35.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-02-22T02:34:35.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-02-22T02:34:35.490 INFO:tasks.workunit.client.0.smithi046.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-02-22T02:34:35.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2024-02-22T02:34:35.491 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-02-22T02:34:35.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-02-22T02:34:35.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-02-22T02:34:35.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2024-02-22T02:34:35.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-02-22T02:34:35.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-02-22T02:34:35.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2024-02-22T02:34:35.927 INFO:tasks.workunit.client.0.smithi046.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-02-22T02:34:35.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-02-22T02:34:35.928 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2024-02-22T02:34:35.928 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-02-22T02:34:35.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-02-22T02:34:35.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2024-02-22T02:34:35.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-02-22T02:34:35.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-02-22T02:34:35.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-02-22T02:34:35.933 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-02-22T02:34:35.933 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:34:35.933 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:34:35.934 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:34:35.934 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:35.934 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:35.934 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:34:35.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-02-22T02:34:35.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-02-22T02:34:36.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2024-02-22T02:34:36.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-02-22T02:34:36.055 INFO:tasks.workunit.client.0.smithi046.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-02-22T02:34:36.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2024-02-22T02:34:36.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-02-22T02:34:36.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2024-02-22T02:34:36.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-02-22T02:34:36.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-02-22T02:34:36.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-02-22T02:34:36.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-02-22T02:34:36.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-02-22T02:34:36.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-02-22T02:34:36.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-02-22T02:34:36.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-02-22T02:34:36.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:34:36.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:34:36.061 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:34:36.061 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:36.061 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:36.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:34:36.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-02-22T02:34:36.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-02-22T02:34:36.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2024-02-22T02:34:36.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2024-02-22T02:34:36.180 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-02-22T02:34:36.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2024-02-22T02:34:36.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-02-22T02:34:36.863 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-02-22T02:34:36.864 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-02-22T02:34:36.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-02-22T02:34:36.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-02-22T02:34:36.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-02-22T02:34:38.867 INFO:tasks.workunit.client.0.smithi046.stdout:Check configuration values - Attempt#: 0 2024-02-22T02:34:38.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-02-22T02:34:38.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-02-22T02:34:39.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-02-22T02:34:39.302 INFO:tasks.workunit.client.0.smithi046.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-02-22T02:34:39.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-02-22T02:34:39.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-02-22T02:34:39.303 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-02-22T02:34:39.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-02-22T02:34:39.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-02-22T02:34:39.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-02-22T02:34:39.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-02-22T02:34:39.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-02-22T02:34:39.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2024-02-22T02:34:39.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-02-22T02:34:39.714 INFO:tasks.workunit.client.0.smithi046.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-02-22T02:34:39.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2024-02-22T02:34:39.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-02-22T02:34:39.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-02-22T02:34:39.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2024-02-22T02:34:39.719 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-02-22T02:34:39.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-02-22T02:34:39.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-02-22T02:34:39.720 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-02-22T02:34:39.720 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:34:39.720 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:34:39.721 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:34:39.721 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:39.721 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:39.721 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:34:39.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-02-22T02:34:39.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-02-22T02:34:39.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-02-22T02:34:39.845 INFO:tasks.workunit.client.0.smithi046.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-02-22T02:34:39.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-02-22T02:34:39.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-02-22T02:34:39.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-02-22T02:34:39.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-02-22T02:34:39.848 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-02-22T02:34:39.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-02-22T02:34:39.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-02-22T02:34:39.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-02-22T02:34:39.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2024-02-22T02:34:39.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:34:39.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:34:39.851 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:34:39.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:34:39.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:34:39.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:34:39.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:34:39.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:34:40.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:34:40.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:34:40.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:34:40.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:34:40.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:34:40.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:34:40.166 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:34:40.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:34:40.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:34:40.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:34:40.167 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:34:40.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:34:40.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:34:40.170 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:34:40.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:34:40.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:40.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:40.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:34:40.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:34:40.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:34:40.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-02-22T02:34:40.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:34:40.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:34:40.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:34:40.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:34:40.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:34:40.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:34:40.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:34:40.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:34:40.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:34:40.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:34:40.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:34:40.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:34:40.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:34:40.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:34:40.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:34:40.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:34:40.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:34:40.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:34:40.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:34:40.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:34:40.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:34:40.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:34:40.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:34:40.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:40.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:40.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:34:40.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:34:40.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:34:40.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-02-22T02:34:40.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-02-22T02:34:40.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-02-22T02:34:40.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-02-22T02:34:40.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:34:40.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:34:40.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:34:40.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:34:40.195 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:34:40.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:34:40.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:34:40.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:34:40.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:34:40.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:34:40.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:34:40.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:34:40.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:34:40.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:34:40.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:34:40.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:34:40.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:34:40.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:34:40.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:34:40.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:34:40.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:34:40.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:34:40.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:34:40.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:40.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:40.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:34:40.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:34:40.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:34:40.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-02-22T02:34:40.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:34:40.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:40.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:40.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20104 2024-02-22T02:34:40.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:34:40.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:34:40.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:34:40.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-02-22T02:34:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2024-02-22T02:34:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2024-02-22T02:34:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2024-02-22T02:34:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-02-22T02:34:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-02-22T02:34:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:34:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:34:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:34:40.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:34:40.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:34:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:34:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:34:40.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:34:40.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:34:40.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:34:40.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:34:40.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:34:40.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:34:40.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:34:40.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:34:40.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:34:40.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:34:40.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:34:40.221 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:34:40.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:34:40.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:34:40.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:34:40.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:34:40.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:40.225 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:40.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:34:40.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:34:40.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:34:40.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-02-22T02:34:40.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:34:40.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:40.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:40.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20104 2024-02-22T02:34:40.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:34:40.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:34:40.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:34:40.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-02-22T02:34:40.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2024-02-22T02:34:40.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-02-22T02:34:40.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:34:40.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:34:40.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:34:40.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-02-22T02:34:40.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-02-22T02:34:40.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:34:40.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:34:40.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:34:40.604 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:34:40.604 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:40.604 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:40.604 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:34:40.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-22T02:34:40.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:34:40.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:34:40.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:34:40.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:34:40.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:34:40.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:34:40.834 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:34:40.834 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:34:40.834 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:34:40.834 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:34:40.834 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:40.835 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:40.835 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-mon.a.asok 2024-02-22T02:34:40.835 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:34:40.835 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20104/ceph-mon.a.asok config get fsid 2024-02-22T02:34:40.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:34:40.957 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:34:40.957 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:34:40.957 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:34:40.957 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:34:40.958 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:34:40.958 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:34:40.958 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:34:40.958 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:34:40.958 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:40.959 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:40.959 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-mon.a.asok 2024-02-22T02:34:40.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:34:40.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20104/ceph-mon.a.asok config get mon_host 2024-02-22T02:34:41.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2024-02-22T02:34:41.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-02-22T02:34:41.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:34:41.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:34:41.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:34:41.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-02-22T02:34:41.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:34:41.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:34:41.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:34:41.973 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:34:41.973 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:34:41.974 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:41.974 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:41.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:34:41.974 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:34:41.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:34:42.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-02-22T02:34:42.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-02-22T02:34:42.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:34:42.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:34:42.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:34:42.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-02-22T02:34:42.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-02-22T02:34:42.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:34:42.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:34:42.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:34:42.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-02-22T02:34:42.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-02-22T02:34:42.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:34:42.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:34:42.013 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-02-22T02:34:42.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:34:42.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:34:42.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:34:42.014 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:34:42.014 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:34:42.014 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:34:42.014 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:34:42.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:34:42.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:34:42.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:34:42.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:34:42.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-02-22T02:34:42.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-02-22T02:34:42.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:34:42.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:34:42.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:34:42.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:34:42.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:34:42.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-02-22T02:34:42.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-02-22T02:34:42.015 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:34:42.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2601a3ac-9c67-4dc2-ac0d-67561d924a89 2024-02-22T02:34:42.016 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 2601a3ac-9c67-4dc2-ac0d-67561d924a89 2024-02-22T02:34:42.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2601a3ac-9c67-4dc2-ac0d-67561d924a89' 2024-02-22T02:34:42.016 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:34:42.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCstZlU6zCARAAqJhZbIa/boo78T+K/+ZOMw== 2024-02-22T02:34:42.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCstZlU6zCARAAqJhZbIa/boo78T+K/+ZOMw=="}' 2024-02-22T02:34:42.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2601a3ac-9c67-4dc2-ac0d-67561d924a89 -i td/mclock-config/0/new.json 2024-02-22T02:34:42.343 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:34:42.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-02-22T02:34:42.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDCstZlU6zCARAAqJhZbIa/boo78T+K/+ZOMw== --osd-uuid 2601a3ac-9c67-4dc2-ac0d-67561d924a89 2024-02-22T02:34:42.396 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:42.391+0000 7fbda4be67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:34:42.420 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:42.415+0000 7fbda4be67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:34:42.421 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:42.419+0000 7fbda4be67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:34:42.422 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:42.419+0000 7fbda4be67c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:34:42.422 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:42.419+0000 7fbda4be67c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:34:42.422 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:42.419+0000 7fbda4be67c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:34:42.451 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:42.447+0000 7fbda4be67c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-02-22T02:34:48.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-02-22T02:34:48.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:34:48.924 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:34:48.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:34:48.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:34:49.481 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:34:49.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:34:49.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-02-22T02:34:49.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:34:49.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:34:49.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:34:49.514 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:49.511+0000 7f8696cb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:34:49.526 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:49.523+0000 7f8696cb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:34:49.527 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:49.523+0000 7f8696cb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:34:49.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:34:49.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:34:49.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:34:49.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:34:49.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:34:49.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:49.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:34:49.900 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:34:49.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:49.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:50.106 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:50.103+0000 7f8696cb97c0 -1 Falling back to public interface 2024-02-22T02:34:50.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:51.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:51.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:51.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:34:51.325 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:34:51.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:51.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:51.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:52.715 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:34:52.711+0000 7f8696cb97c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:34:52.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:52.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:52.750 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:34:52.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:34:52.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:52.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:53.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:54.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:54.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:54.371 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:34:54.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:34:54.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:54.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:54.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:55.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:55.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:55.824 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:34:55.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:34:55.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:55.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:56.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:57.245 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:57.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:57.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:34:57.246 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:34:57.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:57.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:57.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:34:58.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:34:58.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:34:58.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:34:58.815 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:34:58.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:34:58.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:34:59.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:00.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:00.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:00.258 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:35:00.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:35:00.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:00.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:00.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:01.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:01.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:01.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:35:01.826 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:35:01.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:01.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:02.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:03.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:03.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:03.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:35:03.257 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:35:03.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:03.258 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:03.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:04.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:04.833 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:35:04.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:35:04.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:04.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:05.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:06.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:06.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:06.303 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:35:06.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:35:06.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:06.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:07.012 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:08.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:08.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:08.014 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:35:08.014 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:35:08.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:08.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:08.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:09.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:09.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:09.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:35:09.829 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:35:09.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:09.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:10.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:11.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:11.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:11.644 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:35:11.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:35:11.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:11.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:12.519 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:13.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:13.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:13.521 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:35:13.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:35:13.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:13.522 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:14.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:15.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:15.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:15.680 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:35:15.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:35:15.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:15.681 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:16.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:17.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:17.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:17.611 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:35:17.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:35:17.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:17.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:18.109 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3939851906,v1:127.0.0.1:6803/3939851906] [v2:127.0.0.1:6804/3939851906,v1:127.0.0.1:6805/3939851906] exists,up 2601a3ac-9c67-4dc2-ac0d-67561d924a89 2024-02-22T02:35:18.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:35:18.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:35:18.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:35:18.110 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2024-02-22T02:35:18.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2024-02-22T02:35:18.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2024-02-22T02:35:18.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2024-02-22T02:35:18.545 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-02-22T02:35:18.545 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:35:18.545 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:35:18.545 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:35:18.545 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:18.545 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:18.545 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:35:18.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-02-22T02:35:18.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_profile 2024-02-22T02:35:18.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2024-02-22T02:35:18.656 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2024-02-22T02:35:18.656 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2024-02-22T02:35:18.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2024-02-22T02:35:18.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2024-02-22T02:35:18.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-02-22T02:35:18.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2024-02-22T02:35:18.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2024-02-22T02:35:18.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2024-02-22T02:35:18.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-02-22T02:35:18.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-02-22T02:35:18.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-02-22T02:35:18.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-02-22T02:35:18.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:35:18.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:35:18.658 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:35:18.658 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:18.658 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:18.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:35:18.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-02-22T02:35:18.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-02-22T02:35:18.766 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2024-02-22T02:35:18.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2024-02-22T02:35:18.767 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-02-22T02:35:18.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2024-02-22T02:35:18.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2024-02-22T02:35:18.917 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:35:18.917 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '' " 2024-02-22T02:35:18.917 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:35:18.932 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-02-22T02:35:18.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-02-22T02:35:18.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-02-22T02:35:18.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-02-22T02:35:18.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-02-22T02:35:20.935 INFO:tasks.workunit.client.0.smithi046.stdout:Check configuration values - Attempt#: 0 2024-02-22T02:35:20.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-02-22T02:35:20.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-02-22T02:35:21.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-02-22T02:35:21.369 INFO:tasks.workunit.client.0.smithi046.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-02-22T02:35:21.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-02-22T02:35:21.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2024-02-22T02:35:21.370 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-02-22T02:35:21.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-02-22T02:35:21.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-02-22T02:35:21.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2024-02-22T02:35:21.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-02-22T02:35:21.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-02-22T02:35:21.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2024-02-22T02:35:21.786 INFO:tasks.workunit.client.0.smithi046.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-02-22T02:35:21.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-02-22T02:35:21.786 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2024-02-22T02:35:21.787 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-02-22T02:35:21.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-02-22T02:35:21.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2024-02-22T02:35:21.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-02-22T02:35:21.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-02-22T02:35:21.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-02-22T02:35:21.794 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-02-22T02:35:21.794 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:35:21.794 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:35:21.794 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:35:21.794 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:21.794 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:21.794 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:35:21.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-02-22T02:35:21.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-02-22T02:35:21.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2024-02-22T02:35:21.917 INFO:tasks.workunit.client.0.smithi046.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-02-22T02:35:21.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-02-22T02:35:21.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2024-02-22T02:35:21.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-02-22T02:35:21.920 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-02-22T02:35:21.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2024-02-22T02:35:21.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-02-22T02:35:21.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-02-22T02:35:21.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-02-22T02:35:21.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-02-22T02:35:21.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-02-22T02:35:21.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-02-22T02:35:21.925 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-02-22T02:35:21.925 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:35:21.926 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:35:21.926 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:35:21.926 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:21.926 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:21.926 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:35:21.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-02-22T02:35:21.926 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-02-22T02:35:22.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2024-02-22T02:35:22.044 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2024-02-22T02:35:22.044 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-02-22T02:35:22.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2024-02-22T02:35:22.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2024-02-22T02:35:22.205 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:35:22.205 INFO:tasks.workunit.client.0.smithi046.stdout: "success": "osd_mclock_scheduler_client_res = '' " 2024-02-22T02:35:22.205 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:35:22.219 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-02-22T02:35:22.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-02-22T02:35:22.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-02-22T02:35:22.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-02-22T02:35:22.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-02-22T02:35:24.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-02-22T02:35:24.223 INFO:tasks.workunit.client.0.smithi046.stdout:Check configuration values - Attempt#: 0 2024-02-22T02:35:24.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-02-22T02:35:24.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-02-22T02:35:24.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-02-22T02:35:24.646 INFO:tasks.workunit.client.0.smithi046.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-02-22T02:35:24.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2024-02-22T02:35:24.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-02-22T02:35:24.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-02-22T02:35:24.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-02-22T02:35:24.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2024-02-22T02:35:24.650 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-02-22T02:35:24.651 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-02-22T02:35:25.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2024-02-22T02:35:25.078 INFO:tasks.workunit.client.0.smithi046.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-02-22T02:35:25.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-02-22T02:35:25.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2024-02-22T02:35:25.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-02-22T02:35:25.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2024-02-22T02:35:25.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-02-22T02:35:25.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-02-22T02:35:25.084 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-02-22T02:35:25.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-02-22T02:35:25.085 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-02-22T02:35:25.085 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:35:25.085 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:35:25.085 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:35:25.085 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:25.085 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:25.085 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:35:25.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-02-22T02:35:25.085 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-02-22T02:35:25.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2024-02-22T02:35:25.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-02-22T02:35:25.199 INFO:tasks.workunit.client.0.smithi046.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-02-22T02:35:25.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2024-02-22T02:35:25.199 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-02-22T02:35:25.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2024-02-22T02:35:25.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-02-22T02:35:25.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-02-22T02:35:25.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-02-22T02:35:25.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-02-22T02:35:25.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2024-02-22T02:35:25.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:35:25.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:35:25.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:35:25.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:35:25.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:35:25.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:35:25.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:35:25.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:35:25.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:35:25.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:35:25.313 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:35:25.313 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:35:25.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:35:25.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:35:25.315 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:35:25.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:35:25.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:35:25.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:35:25.317 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:35:25.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:35:25.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:35:25.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:35:25.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:35:25.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:25.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:25.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:35:25.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:35:25.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:35:25.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-02-22T02:35:25.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:35:25.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:35:25.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:35:25.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:35:25.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:35:25.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:35:25.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:35:25.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:35:25.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:35:25.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:35:25.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:35:25.332 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:35:25.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:35:25.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:35:25.333 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:35:25.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:35:25.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:35:25.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:35:25.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:35:25.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:35:25.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:35:25.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:35:25.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:35:25.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:25.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:25.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:35:25.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:35:25.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:35:25.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-02-22T02:35:25.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-02-22T02:35:25.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-02-22T02:35:25.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-02-22T02:35:25.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:35:25.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:35:25.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:35:25.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:35:25.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:35:25.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:35:25.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:35:25.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:35:25.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:35:25.342 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:35:25.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:35:25.344 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:35:25.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:35:25.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:35:25.345 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:35:25.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:35:25.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:35:25.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:35:25.347 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:35:25.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:35:25.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:35:25.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:35:25.350 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:35:25.351 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:25.351 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:25.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:35:25.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:35:25.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:35:25.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-02-22T02:35:25.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:35:25.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:25.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:25.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20104 2024-02-22T02:35:25.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:35:25.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:35:25.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:35:25.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-02-22T02:35:25.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2024-02-22T02:35:25.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2024-02-22T02:35:25.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2024-02-22T02:35:25.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-02-22T02:35:25.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-02-22T02:35:25.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:35:25.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:35:25.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:35:25.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:35:25.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:35:25.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:35:25.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:35:25.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:35:25.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:35:25.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:35:25.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:35:25.359 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:35:25.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:35:25.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:35:25.360 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:35:25.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:35:25.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:35:25.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:35:25.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:35:25.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:35:25.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:35:25.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:35:25.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:35:25.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:25.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:25.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:35:25.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:35:25.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:35:25.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-02-22T02:35:25.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:35:25.367 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:25.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:25.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20104 2024-02-22T02:35:25.369 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:35:25.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:35:25.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:35:25.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-02-22T02:35:25.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2024-02-22T02:35:25.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-02-22T02:35:25.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:35:25.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:35:25.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:35:25.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-02-22T02:35:25.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-02-22T02:35:25.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:35:25.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:35:25.780 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:35:25.781 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:35:25.781 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:25.781 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:25.781 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:35:25.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-22T02:35:25.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:35:25.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:35:25.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:35:25.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:35:25.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:35:25.999 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:35:26.000 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:35:26.000 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:35:26.000 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:35:26.000 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:35:26.001 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:26.001 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:26.001 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-mon.a.asok 2024-02-22T02:35:26.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:35:26.001 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20104/ceph-mon.a.asok config get fsid 2024-02-22T02:35:26.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:35:26.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:35:26.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:35:26.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:35:26.118 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:35:26.119 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:35:26.119 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:35:26.119 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:35:26.119 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:35:26.120 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:26.120 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:26.120 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-mon.a.asok 2024-02-22T02:35:26.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:35:26.120 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20104/ceph-mon.a.asok config get mon_host 2024-02-22T02:35:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-02-22T02:35:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-02-22T02:35:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:35:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:35:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:35:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-02-22T02:35:26.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:35:27.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:35:27.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:35:27.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:35:27.134 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:35:27.134 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:27.134 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:27.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:35:27.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:35:27.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:35:27.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-02-22T02:35:27.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-02-22T02:35:27.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:35:27.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:35:27.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:35:27.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-02-22T02:35:27.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-02-22T02:35:27.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:35:27.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:35:27.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:35:27.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-02-22T02:35:27.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-02-22T02:35:27.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:35:27.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:35:27.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-02-22T02:35:27.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:35:27.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:35:27.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:35:27.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:35:27.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:35:27.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:35:27.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:35:27.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' 2024-02-22T02:35:27.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:35:27.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:35:27.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:35:27.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-02-22T02:35:27.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-02-22T02:35:27.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:35:27.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:35:27.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:35:27.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:35:27.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:35:27.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-02-22T02:35:27.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-02-22T02:35:27.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:35:27.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7d46a83e-09f6-41dd-86dc-80057f93652c 2024-02-22T02:35:27.212 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 7d46a83e-09f6-41dd-86dc-80057f93652c 2024-02-22T02:35:27.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7d46a83e-09f6-41dd-86dc-80057f93652c' 2024-02-22T02:35:27.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:35:27.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvstZlGdiJDRAAKadTudQjQQt2qUxhA3xoaQ== 2024-02-22T02:35:27.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvstZlGdiJDRAAKadTudQjQQt2qUxhA3xoaQ=="}' 2024-02-22T02:35:27.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7d46a83e-09f6-41dd-86dc-80057f93652c -i td/mclock-config/0/new.json 2024-02-22T02:35:27.528 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:35:27.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-02-22T02:35:27.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDvstZlGdiJDRAAKadTudQjQQt2qUxhA3xoaQ== --osd-uuid 7d46a83e-09f6-41dd-86dc-80057f93652c 2024-02-22T02:35:27.582 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:35:27.574+0000 7f5ca066d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:35:27.596 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:35:27.594+0000 7f5ca066d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:35:27.602 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:35:27.594+0000 7f5ca066d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:35:27.602 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:35:27.594+0000 7f5ca066d7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:35:27.602 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:35:27.594+0000 7f5ca066d7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:35:27.602 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:35:27.594+0000 7f5ca066d7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-02-22T02:35:27.636 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:35:27.634+0000 7f5ca066d7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-02-22T02:35:34.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-02-22T02:35:34.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:35:34.023 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:35:34.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:35:34.024 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:35:34.598 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:35:34.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:35:34.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3ae18c79-3296-4d3e-80d9-f945b613d086 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20104/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-02-22T02:35:34.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:35:34.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:35:34.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:35:34.631 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:35:34.626+0000 7f0f5ea987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:35:34.659 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:35:34.654+0000 7f0f5ea987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:35:34.661 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:35:34.658+0000 7f0f5ea987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:35:35.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:35:35.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:35:35.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:35:35.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:35:35.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:35:35.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:35.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:35:35.075 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:35:35.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:35.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:35.246 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:35:35.242+0000 7f0f5ea987c0 -1 Falling back to public interface 2024-02-22T02:35:35.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:36.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:36.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:36.513 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:35:36.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:35:36.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:36.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:36.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:37.882 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:35:37.878+0000 7f0f5ea987c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:35:37.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:37.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:37.946 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:35:37.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:35:37.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:37.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:38.607 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:39.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:39.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:39.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:35:39.609 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:35:39.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:39.610 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:40.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:41.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:41.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:41.051 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:35:41.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:35:41.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:41.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:41.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:42.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:42.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:42.478 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:35:42.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:35:42.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:42.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:43.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:44.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:44.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:44.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:35:44.037 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:35:44.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:44.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:44.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:45.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:45.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:45.513 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:35:45.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:35:45.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:45.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:46.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:47.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:47.086 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:35:47.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:47.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:35:47.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:47.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:47.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:48.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:48.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:48.540 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:35:48.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:35:48.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:48.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:49.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:50.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:50.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:50.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:35:50.086 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:35:50.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:50.087 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:50.532 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:51.534 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:51.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:51.535 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:35:51.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:35:51.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:51.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:52.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:53.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:53.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:53.337 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:35:53.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:35:53.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:53.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:54.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:55.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:55.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:55.171 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:35:55.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:35:55.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:55.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:55.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:56.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:56.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:56.963 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:35:56.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:35:56.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:56.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:57.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:35:58.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:35:58.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:35:58.963 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:35:58.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:35:58.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:35:58.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:35:59.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:00.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:00.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:00.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:36:00.749 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:36:00.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:00.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:01.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:02.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:02.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:02.812 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:36:02.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:36:02.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:02.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:03.253 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2752959585,v1:127.0.0.1:6803/2752959585] [v2:127.0.0.1:6804/2752959585,v1:127.0.0.1:6805/2752959585] exists,up 7d46a83e-09f6-41dd-86dc-80057f93652c 2024-02-22T02:36:03.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:36:03.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:36:03.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:36:03.255 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-02-22T02:36:03.255 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:36:03.255 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:36:03.256 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:36:03.256 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:03.256 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:36:03.256 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:36:03.256 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-02-22T02:36:03.257 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_recovery_max_active 2024-02-22T02:36:03.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2024-02-22T02:36:03.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2024-02-22T02:36:03.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2024-02-22T02:36:03.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-02-22T02:36:04.016 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-02-22T02:36:06.019 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-02-22T02:36:06.020 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:36:06.020 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:36:06.020 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:36:06.021 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:06.021 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:36:06.021 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:36:06.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-02-22T02:36:06.021 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_recovery_max_active 2024-02-22T02:36:06.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2024-02-22T02:36:06.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2024-02-22T02:36:06.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-02-22T02:36:06.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-02-22T02:36:07.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-02-22T02:36:09.218 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-02-22T02:36:09.218 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:36:09.218 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:36:09.219 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:36:09.219 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:09.219 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:36:09.219 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20104/ceph-osd.0.asok 2024-02-22T02:36:09.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-02-22T02:36:09.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20104/ceph-osd.0.asok config get osd_recovery_max_active 2024-02-22T02:36:09.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2024-02-22T02:36:09.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2024-02-22T02:36:09.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2024-02-22T02:36:09.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:36:09.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:36:09.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:36:09.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:36:09.354 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:36:09.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:36:09.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:36:09.354 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:36:09.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:36:09.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:36:09.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:36:09.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:36:09.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:36:09.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:36:09.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:36:09.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:36:09.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:36:09.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:36:09.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:36:09.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:36:09.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:36:09.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:36:09.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:36:09.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:09.681 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:36:09.682 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:36:09.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:36:09.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:36:09.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-02-22T02:36:09.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:36:09.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:36:09.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:36:09.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:36:09.685 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:36:09.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:36:09.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:36:09.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:36:09.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:36:09.687 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:36:09.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:36:09.689 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:36:09.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:36:09.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:36:09.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:36:09.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:36:09.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:36:09.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:36:09.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:36:09.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:36:09.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:36:09.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:36:09.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:36:09.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:09.698 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:36:09.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:36:09.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:36:09.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:36:09.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-22T02:36:09.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/mclock-config 0 2024-02-22T02:36:09.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-02-22T02:36:09.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-22T02:36:09.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-02-22T02:36:09.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:36:09.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:36:09.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:36:09.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:36:09.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:36:09.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:36:09.703 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:36:09.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:36:09.704 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:36:09.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:36:09.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:36:09.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:36:09.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:36:09.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:36:09.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:36:09.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:36:09.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:36:09.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-22T02:36:09.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-02-22T02:36:09.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:36:09.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:09.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20104 2024-02-22T02:36:09.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20104 2024-02-22T02:36:09.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:36:09.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:36:09.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-22T02:36:09.715 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-22T02:36:09.716 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-22T02:36:09.769 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2024-02-22T02:36:09.769 DEBUG:teuthology.orchestra.run.smithi046:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbe7fcec16eddee6239e4dd68c4d0203c7df6461 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2024-02-22T02:36:09.825 INFO:tasks.workunit.client.0.smithi046.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-22T02:36:09.831 INFO:tasks.workunit.client.0.smithi046.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-22T02:36:09.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-22T02:36:09.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-22T02:36:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-22T02:36:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-22T02:36:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-22T02:36:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-22T02:36:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-22T02:36:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-22T02:36:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/network-ping 2024-02-22T02:36:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2024-02-22T02:36:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2024-02-22T02:36:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2024-02-22T02:36:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2024-02-22T02:36:09.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2024-02-22T02:36:09.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2024-02-22T02:36:09.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none ' 2024-02-22T02:36:09.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-02-22T02:36:09.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2024-02-22T02:36:09.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2024-02-22T02:36:09.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2024-02-22T02:36:09.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2024-02-22T02:36:09.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-22T02:36:09.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2024-02-22T02:36:09.837 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_network_ping_test2' 2024-02-22T02:36:09.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-02-22T02:36:09.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-02-22T02:36:09.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-02-22T02:36:09.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-02-22T02:36:09.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-02-22T02:36:09.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:36:09.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-02-22T02:36:09.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:36:09.837 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:36:09.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:36:09.837 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:36:09.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:36:09.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:36:09.839 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:36:09.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:36:09.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:36:09.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:36:09.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:36:09.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:36:09.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:36:09.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:36:09.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:36:09.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:36:09.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:36:09.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:36:09.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-02-22T02:36:09.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:36:09.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:09.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:36:09.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34135 2024-02-22T02:36:09.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:36:09.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:36:09.849 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-02-22T02:36:09.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:36:09.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:09.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:36:09.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.34135 2024-02-22T02:36:09.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:36:09.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-22T02:36:09.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-22T02:36:09.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:36:09.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-02-22T02:36:09.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2024-02-22T02:36:09.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2024-02-22T02:36:09.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2024-02-22T02:36:09.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-02-22T02:36:09.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:36:09.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:36:09.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:36:09.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-02-22T02:36:09.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-02-22T02:36:10.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:36:10.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:36:10.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:36:10.276 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:36:10.276 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:10.276 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:36:10.276 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:36:10.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-22T02:36:10.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:36:10.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:36:10.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:36:10.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:36:10.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:36:10.494 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:36:10.494 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:36:10.494 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:36:10.495 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:36:10.495 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:36:10.495 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:10.495 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:36:10.495 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-mon.a.asok 2024-02-22T02:36:10.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:36:10.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.34135/ceph-mon.a.asok config get fsid 2024-02-22T02:36:10.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:36:10.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:36:10.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:36:10.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:36:10.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:36:10.616 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:36:10.616 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:36:10.616 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:36:10.617 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:36:10.617 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:10.617 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:36:10.617 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-mon.a.asok 2024-02-22T02:36:10.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:36:10.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.34135/ceph-mon.a.asok config get mon_host 2024-02-22T02:36:10.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2024-02-22T02:36:10.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-02-22T02:36:10.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:36:10.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:36:10.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:36:10.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-02-22T02:36:10.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:36:11.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:36:11.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:36:11.652 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:36:11.653 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:36:11.653 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:11.653 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:36:11.653 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:36:11.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:36:11.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:36:11.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2024-02-22T02:36:11.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-02-22T02:36:11.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:36:11.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:36:11.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:36:11.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-02-22T02:36:11.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-02-22T02:36:11.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:36:11.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:36:11.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:36:11.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-02-22T02:36:11.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-02-22T02:36:11.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-02-22T02:36:11.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-02-22T02:36:11.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:36:11.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:36:11.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:36:11.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:36:11.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:36:11.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:36:11.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-02-22T02:36:11.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:36:11.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dac716f8-e1c2-4fdc-8270-92a8aab277c8 2024-02-22T02:36:11.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dac716f8-e1c2-4fdc-8270-92a8aab277c8' 2024-02-22T02:36:11.732 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 dac716f8-e1c2-4fdc-8270-92a8aab277c8 2024-02-22T02:36:11.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:36:11.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbs9ZlML2oLBAAGbBmWLm4fCDlSVJFC5VYZQ== 2024-02-22T02:36:11.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbs9ZlML2oLBAAGbBmWLm4fCDlSVJFC5VYZQ=="}' 2024-02-22T02:36:11.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dac716f8-e1c2-4fdc-8270-92a8aab277c8 -i td/network-ping/0/new.json 2024-02-22T02:36:12.065 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:36:12.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-02-22T02:36:12.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbs9ZlML2oLBAAGbBmWLm4fCDlSVJFC5VYZQ== --osd-uuid dac716f8-e1c2-4fdc-8270-92a8aab277c8 2024-02-22T02:36:12.120 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:12.114+0000 7f2e5d2ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:36:12.151 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:12.146+0000 7f2e5d2ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:36:12.152 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:12.150+0000 7f2e5d2ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:36:12.153 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:12.150+0000 7f2e5d2ec7c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-02-22T02:36:12.153 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:12.150+0000 7f2e5d2ec7c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-02-22T02:36:12.153 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:12.150+0000 7f2e5d2ec7c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-02-22T02:36:12.182 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:12.178+0000 7f2e5d2ec7c0 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-02-22T02:36:18.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-02-22T02:36:18.639 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:36:18.640 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:36:18.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:36:18.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:36:19.200 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:36:19.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:36:19.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:36:19.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:36:19.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:36:19.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:36:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:19.230+0000 7f1c1a7b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:36:19.240 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:19.234+0000 7f1c1a7b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:36:19.242 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:19.238+0000 7f1c1a7b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:36:19.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:36:19.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:36:19.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:36:19.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:36:19.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:36:19.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:19.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:36:19.620 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:36:19.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:19.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:19.811 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:19.806+0000 7f1c1a7b57c0 -1 Falling back to public interface 2024-02-22T02:36:20.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:21.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:21.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:21.051 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:36:21.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:36:21.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:21.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:21.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:22.392 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:22.386+0000 7f1c1a7b57c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:36:22.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:22.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:22.473 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:36:22.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:36:22.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:22.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:23.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:24.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:24.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:24.041 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:36:24.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:36:24.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:24.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:24.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:25.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:25.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:25.474 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:36:25.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:36:25.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:25.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:25.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:26.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:26.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:26.972 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:36:26.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:36:26.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:26.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:27.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:28.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:28.486 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:36:28.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:28.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:36:28.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:28.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:28.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:29.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:29.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:29.920 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:36:29.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:36:29.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:29.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:30.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:31.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:31.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:31.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:36:31.457 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:36:31.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:31.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:31.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:32.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:32.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:32.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:36:32.944 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:36:32.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:32.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:33.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:34.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:34.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:34.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:36:34.503 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:36:34.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:34.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:34.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:35.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:35.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:35.943 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:36:35.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:36:35.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:35.944 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:36.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:37.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:37.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:37.690 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:36:37.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:36:37.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:37.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:38.562 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:39.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:39.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:39.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:36:39.565 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:36:39.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:39.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:40.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:41.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:41.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:41.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:36:41.324 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:36:41.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:41.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:42.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:43.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:43.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:43.299 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:36:43.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:36:43.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:43.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:44.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:45.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:45.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:45.510 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:36:45.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:36:45.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:45.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:46.426 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:47.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:47.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:47.428 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:36:47.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:36:47.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:47.429 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:36:47.893 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1137811377,v1:127.0.0.1:6803/1137811377] [v2:127.0.0.1:6804/1137811377,v1:127.0.0.1:6805/1137811377] exists,up dac716f8-e1c2-4fdc-8270-92a8aab277c8 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:36:47.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:36:47.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:36:47.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-02-22T02:36:47.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-02-22T02:36:47.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:36:47.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:36:47.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:36:47.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:36:47.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:36:47.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:36:47.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-02-22T02:36:47.899 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:36:47.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8cc44149-cf31-4668-b4c4-7c8b140d31db 2024-02-22T02:36:47.900 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 8cc44149-cf31-4668-b4c4-7c8b140d31db 2024-02-22T02:36:47.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8cc44149-cf31-4668-b4c4-7c8b140d31db' 2024-02-22T02:36:47.900 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:36:47.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/s9ZlQSWyNhAASDYV824SUgWS4OJ7QpW2Aw== 2024-02-22T02:36:47.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/s9ZlQSWyNhAASDYV824SUgWS4OJ7QpW2Aw=="}' 2024-02-22T02:36:47.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8cc44149-cf31-4668-b4c4-7c8b140d31db -i td/network-ping/1/new.json 2024-02-22T02:36:48.462 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:36:48.477 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-02-22T02:36:48.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/s9ZlQSWyNhAASDYV824SUgWS4OJ7QpW2Aw== --osd-uuid 8cc44149-cf31-4668-b4c4-7c8b140d31db 2024-02-22T02:36:48.515 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:48.510+0000 7f09f3fe97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:36:48.566 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:48.562+0000 7f09f3fe97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:36:48.568 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:48.562+0000 7f09f3fe97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:36:48.568 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:48.566+0000 7f09f3fe97c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-02-22T02:36:48.568 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:48.566+0000 7f09f3fe97c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-02-22T02:36:48.569 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:48.566+0000 7f09f3fe97c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-02-22T02:36:48.690 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:48.686+0000 7f09f3fe97c0 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-02-22T02:36:55.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-02-22T02:36:55.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:36:55.319 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-02-22T02:36:55.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-22T02:36:55.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:36:55.883 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-02-22T02:36:55.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-22T02:36:55.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:36:55.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:36:55.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:36:55.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:36:55.916 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:55.913+0000 7faa58f277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:36:55.947 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:55.941+0000 7faa58f277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:36:55.948 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:55.945+0000 7faa58f277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:36:56.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-22T02:36:56.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:36:56.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-22T02:36:56.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:36:56.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:36:56.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:56.337 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:36:56.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:36:56.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:36:56.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:56.554 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:56.549+0000 7faa58f277c0 -1 Falling back to public interface 2024-02-22T02:36:56.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:57.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:57.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:57.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:36:57.757 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:36:57.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:57.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:36:58.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:36:59.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:36:59.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:36:59.176 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:36:59.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:36:59.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:36:59.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:36:59.230 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:36:59.225+0000 7faa58f277c0 -1 osd.1 0 log_to_monitors true 2024-02-22T02:36:59.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:00.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:00.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:00.789 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:37:00.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:37:00.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:00.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:01.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:02.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:02.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:02.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:37:02.390 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:37:02.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:02.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:03.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:04.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:04.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:04.031 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:37:04.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:37:04.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:04.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:04.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:05.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:05.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:05.486 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:37:05.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:37:05.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:05.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:05.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:06.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:06.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:06.953 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:37:06.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:37:06.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:06.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:07.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:08.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:08.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:08.381 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:37:08.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:37:08.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:08.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:08.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:09.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:09.872 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:37:09.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:37:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:10.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:11.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:11.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:11.306 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:37:11.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:37:11.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:11.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:11.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:12.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:12.795 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:12.796 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:37:12.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:37:12.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:12.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:13.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:14.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:14.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:14.824 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:37:14.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:37:14.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:14.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:15.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:16.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:16.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:16.674 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:37:16.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:37:16.674 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:16.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:17.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:18.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:18.128 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:37:18.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:18.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:37:18.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:18.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:18.917 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:19.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:19.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:19.920 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:37:19.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:37:19.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:19.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:20.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:21.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:21.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:21.700 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:37:21.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:37:21.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:21.701 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:22.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:23.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:23.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:23.486 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:37:23.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:37:23.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:23.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:24.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:25.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:25.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:25.107 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-02-22T02:37:25.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-22T02:37:25.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:25.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:37:25.550 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/534666511,v1:127.0.0.1:6811/534666511] [v2:127.0.0.1:6812/534666511,v1:127.0.0.1:6813/534666511] exists,up 8cc44149-cf31-4668-b4c4-7c8b140d31db 2024-02-22T02:37:25.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:37:25.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:37:25.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:37:25.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2024-02-22T02:37:25.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-02-22T02:37:25.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:37:25.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-22T02:37:25.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:37:25.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-02-22T02:37:25.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-02-22T02:37:25.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:37:25.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:37:25.553 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:37:25.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-02-22T02:37:25.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-02-22T02:37:25.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:37:25.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:37:25.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-02-22T02:37:25.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:37:25.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:37:25.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:37:25.554 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:37:25.554 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:37:25.554 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:37:25.554 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:37:25.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:37:25.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:37:25.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:37:25.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:37:25.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-02-22T02:37:25.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-02-22T02:37:25.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:37:25.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:37:25.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:37:25.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:37:25.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:37:25.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:37:25.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-02-22T02:37:25.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:37:25.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=248898e7-cb0d-4aec-8ed6-8b3497a0434b 2024-02-22T02:37:25.557 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 248898e7-cb0d-4aec-8ed6-8b3497a0434b 2024-02-22T02:37:25.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 248898e7-cb0d-4aec-8ed6-8b3497a0434b' 2024-02-22T02:37:25.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:37:25.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBls9Zl6zMyIhAAgjnHkvlAQcvtLGmhgeu6AA== 2024-02-22T02:37:25.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBls9Zl6zMyIhAAgjnHkvlAQcvtLGmhgeu6AA=="}' 2024-02-22T02:37:25.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 248898e7-cb0d-4aec-8ed6-8b3497a0434b -i td/network-ping/2/new.json 2024-02-22T02:37:26.099 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:37:26.115 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-02-22T02:37:26.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBls9Zl6zMyIhAAgjnHkvlAQcvtLGmhgeu6AA== --osd-uuid 248898e7-cb0d-4aec-8ed6-8b3497a0434b 2024-02-22T02:37:26.146 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:37:26.141+0000 7f29a79f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:37:26.148 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:37:26.141+0000 7f29a79f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:37:26.149 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:37:26.145+0000 7f29a79f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:37:26.149 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:37:26.145+0000 7f29a79f57c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-02-22T02:37:26.149 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:37:26.145+0000 7f29a79f57c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-02-22T02:37:26.149 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:37:26.145+0000 7f29a79f57c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-02-22T02:37:26.192 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:37:26.185+0000 7f29a79f57c0 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-02-22T02:37:32.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-02-22T02:37:32.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:37:32.910 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-02-22T02:37:32.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-22T02:37:32.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:37:33.472 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-02-22T02:37:33.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-22T02:37:33.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:37:33.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:37:33.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:37:33.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:37:33.508 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:37:33.501+0000 7fef0f40e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:37:33.526 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:37:33.521+0000 7fef0f40e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:37:33.528 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:37:33.521+0000 7fef0f40e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:37:33.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-22T02:37:33.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:37:33.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-22T02:37:33.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:37:33.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:37:33.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:33.913 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:37:33.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:37:33.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:33.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:34.094 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:37:34.089+0000 7fef0f40e7c0 -1 Falling back to public interface 2024-02-22T02:37:34.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:35.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:35.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:35.328 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:37:35.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:37:35.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:35.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:35.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:36.711 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:37:36.705+0000 7fef0f40e7c0 -1 osd.2 0 log_to_monitors true 2024-02-22T02:37:36.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:36.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:36.750 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:37:36.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:37:36.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:36.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:37.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:38.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:38.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:38.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:37:38.351 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:37:38.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:38.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:38.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:39.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:39.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:39.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:37:39.910 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:37:39.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:39.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:40.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:41.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:41.383 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:37:41.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:41.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:37:41.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:41.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:41.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:42.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:42.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:42.832 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:37:42.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:37:42.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:42.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:43.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:44.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:44.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:44.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:37:44.385 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:37:44.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:44.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:44.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:45.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:45.885 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:45.885 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:37:45.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:37:45.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:45.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:46.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:47.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:47.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:47.310 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:37:47.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:37:47.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:47.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:47.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:48.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:48.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:48.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:37:48.810 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:37:48.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:48.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:49.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:50.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:50.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:50.389 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:37:50.389 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:37:50.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:50.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:51.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:52.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:52.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:52.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:37:52.340 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:37:52.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:52.341 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:53.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:54.021 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:54.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:54.022 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:37:54.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:37:54.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:54.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:54.796 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:55.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:55.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:55.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:37:55.799 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:37:55.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:55.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:56.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:57.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:57.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:57.443 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:37:57.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:37:57.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:57.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:37:58.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:37:59.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:37:59.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:37:59.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:37:59.096 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:37:59.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:37:59.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:38:00.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:38:01.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:38:01.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:38:01.214 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:38:01.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:38:01.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:38:01.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:38:01.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:38:02.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:38:02.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:38:02.820 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-02-22T02:38:02.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-22T02:38:02.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:38:02.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:38:03.265 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1647601364,v1:127.0.0.1:6819/1647601364] [v2:127.0.0.1:6820/1647601364,v1:127.0.0.1:6821/1647601364] exists,up 248898e7-cb0d-4aec-8ed6-8b3497a0434b 2024-02-22T02:38:03.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:38:03.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:38:03.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:38:03.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2024-02-22T02:38:08.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2024-02-22T02:38:08.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-02-22T02:38:08.808 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'foo' already exists 2024-02-22T02:38:08.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-22T02:38:09.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-02-22T02:38:09.868 INFO:tasks.workunit.client.0.smithi046.stdout:hints = 1 2024-02-22T02:38:09.868 INFO:tasks.workunit.client.0.smithi046.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-02-22T02:38:09.868 INFO:tasks.workunit.client.0.smithi046.stdout:Object prefix: benchmark_data_smithi046_39677 2024-02-22T02:38:09.868 INFO:tasks.workunit.client.0.smithi046.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-22T02:38:09.868 INFO:tasks.workunit.client.0.smithi046.stdout: 0 0 0 0 0 0 - 0 2024-02-22T02:38:10.868 INFO:tasks.workunit.client.0.smithi046.stdout: 1 16 49 33 0.128891 0.128906 0.270881 0.405104 2024-02-22T02:38:11.868 INFO:tasks.workunit.client.0.smithi046.stdout: 2 16 97 81 0.158181 0.1875 0.283404 0.353266 2024-02-22T02:38:12.868 INFO:tasks.workunit.client.0.smithi046.stdout: 3 16 151 135 0.175759 0.210938 0.310462 0.34918 2024-02-22T02:38:13.868 INFO:tasks.workunit.client.0.smithi046.stdout: 4 16 200 184 0.179665 0.191406 0.260883 0.33171 2024-02-22T02:38:14.868 INFO:tasks.workunit.client.0.smithi046.stdout: 5 16 248 232 0.181227 0.1875 0.283534 0.328218 2024-02-22T02:38:15.868 INFO:tasks.workunit.client.0.smithi046.stdout: 6 16 311 295 0.192032 0.246094 0.280889 0.31974 2024-02-22T02:38:16.869 INFO:tasks.workunit.client.0.smithi046.stdout: 7 16 360 344 0.191938 0.191406 0.308369 0.314298 2024-02-22T02:38:17.869 INFO:tasks.workunit.client.0.smithi046.stdout: 8 16 423 407 0.198703 0.246094 0.260959 0.309758 2024-02-22T02:38:18.869 INFO:tasks.workunit.client.0.smithi046.stdout: 9 16 482 466 0.20223 0.230469 0.284906 0.30565 2024-02-22T02:38:19.869 INFO:tasks.workunit.client.0.smithi046.stdout: 10 16 535 519 0.202707 0.207031 0.285291 0.304916 2024-02-22T02:38:20.869 INFO:tasks.workunit.client.0.smithi046.stdout:Total time run: 10.1986 2024-02-22T02:38:20.869 INFO:tasks.workunit.client.0.smithi046.stdout:Total writes made: 535 2024-02-22T02:38:20.869 INFO:tasks.workunit.client.0.smithi046.stdout:Write size: 4096 2024-02-22T02:38:20.869 INFO:tasks.workunit.client.0.smithi046.stdout:Object size: 4096 2024-02-22T02:38:20.869 INFO:tasks.workunit.client.0.smithi046.stdout:Bandwidth (MB/sec): 0.204915 2024-02-22T02:38:20.870 INFO:tasks.workunit.client.0.smithi046.stdout:Stddev Bandwidth: 0.0346193 2024-02-22T02:38:20.870 INFO:tasks.workunit.client.0.smithi046.stdout:Max bandwidth (MB/sec): 0.246094 2024-02-22T02:38:20.870 INFO:tasks.workunit.client.0.smithi046.stdout:Min bandwidth (MB/sec): 0.128906 2024-02-22T02:38:20.870 INFO:tasks.workunit.client.0.smithi046.stdout:Average IOPS: 52 2024-02-22T02:38:20.870 INFO:tasks.workunit.client.0.smithi046.stdout:Stddev IOPS: 8.86253 2024-02-22T02:38:20.870 INFO:tasks.workunit.client.0.smithi046.stdout:Max IOPS: 63 2024-02-22T02:38:20.870 INFO:tasks.workunit.client.0.smithi046.stdout:Min IOPS: 33 2024-02-22T02:38:20.870 INFO:tasks.workunit.client.0.smithi046.stdout:Average Latency(s): 0.304103 2024-02-22T02:38:20.870 INFO:tasks.workunit.client.0.smithi046.stdout:Stddev Latency(s): 0.061094 2024-02-22T02:38:20.870 INFO:tasks.workunit.client.0.smithi046.stdout:Max latency(s): 0.833717 2024-02-22T02:38:20.870 INFO:tasks.workunit.client.0.smithi046.stdout:Min latency(s): 0.250773 2024-02-22T02:38:20.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2024-02-22T02:38:30.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2024-02-22T02:38:30.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-22T02:38:30.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-22T02:38:31.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-22T02:38:31.414 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-02-22T02:38:31.414 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-02-22T02:38:31.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-22T02:38:31.414 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:38:31.415 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-22T02:38:31.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-02-22T02:38:31.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-02-22T02:38:31.574 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-02-22T02:38:31.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:38:31.575 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-22T02:38:31.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-02-22T02:38:31.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-02-22T02:38:31.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672977' 2024-02-22T02:38:31.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:38:31.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-22T02:38:31.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-02-22T02:38:31.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-02-22T02:38:31.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672977 2-64424509455' 2024-02-22T02:38:31.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:38:31.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-02-22T02:38:31.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:38:31.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-22T02:38:31.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-02-22T02:38:31.880 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:38:31.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-02-22T02:38:31.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-02-22T02:38:31.882 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836507 2024-02-22T02:38:31.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-22T02:38:32.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836507 2024-02-22T02:38:32.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-22T02:38:33.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-22T02:38:33.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-22T02:38:33.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-02-22T02:38:33.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:38:33.726 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-02-22T02:38:33.726 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:38:33.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-22T02:38:33.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:38:33.729 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-02-22T02:38:33.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-02-22T02:38:33.729 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-02-22T02:38:33.730 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672977 2024-02-22T02:38:33.730 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-22T02:38:34.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-02-22T02:38:34.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:38:34.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-02-22T02:38:34.163 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:38:34.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-22T02:38:34.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-02-22T02:38:34.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:38:34.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-02-22T02:38:34.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-02-22T02:38:34.167 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509455 2024-02-22T02:38:34.167 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-22T02:38:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509455 2024-02-22T02:38:34.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:38:34.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2024-02-22T02:38:34.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:38:34.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:38:34.596 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:38:34.596 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:38:34.596 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:38:34.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-osd.0.asok 2024-02-22T02:38:34.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:38:34.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-osd.0.asok dump_osd_network 2024-02-22T02:38:34.699 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:38:34.700 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 1000, 2024-02-22T02:38:34.700 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [] 2024-02-22T02:38:34.700 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:38:34.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:34.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2024-02-22T02:38:34.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2024-02-22T02:38:34.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:34.751 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2024-02-22T02:38:34.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2024-02-22T02:38:34.788 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:38:34.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2024-02-22T02:38:34.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-02-22T02:38:34.790 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-02-22T02:38:34.790 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:38:34.791 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:38:34.791 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:38:34.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-mgr.x.asok 2024-02-22T02:38:34.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:38:34.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-mgr.x.asok dump_osd_network 2024-02-22T02:38:34.899 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:38:34.899 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 1000, 2024-02-22T02:38:34.899 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [] 2024-02-22T02:38:34.899 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:38:34.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:34.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2024-02-22T02:38:34.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2024-02-22T02:38:34.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:34.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2024-02-22T02:38:34.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2024-02-22T02:38:34.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:38:34.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2024-02-22T02:38:34.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:38:34.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:38:34.991 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:38:34.991 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:38:34.991 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:38:34.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-osd.0.asok 2024-02-22T02:38:34.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:38:34.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-osd.0.asok dump_osd_network 0 2024-02-22T02:38:35.097 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:38:35.097 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 0, 2024-02-22T02:38:35.097 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [ 2024-02-22T02:38:35.097 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.097 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:33 2024", 2024-02-22T02:38:35.097 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.097 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:35.097 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:38:35.097 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:35.097 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.097 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.475, 2024-02-22T02:38:35.097 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.596, 2024-02-22T02:38:35.097 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.596 2024-02-22T02:38:35.098 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.098 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.098 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.615, 2024-02-22T02:38:35.098 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.774, 2024-02-22T02:38:35.098 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.774 2024-02-22T02:38:35.098 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.098 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.098 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.615, 2024-02-22T02:38:35.098 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.774, 2024-02-22T02:38:35.098 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.774 2024-02-22T02:38:35.098 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.098 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.417 2024-02-22T02:38:35.098 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:34 2024", 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.455, 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.573, 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.494 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.584, 2024-02-22T02:38:35.099 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 1.380, 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 1.380 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.584, 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 1.380, 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 1.380 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.462 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:33 2024", 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:38:35.100 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:35.101 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.101 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.469, 2024-02-22T02:38:35.101 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.564, 2024-02-22T02:38:35.101 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.533 2024-02-22T02:38:35.101 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.101 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.101 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.554, 2024-02-22T02:38:35.101 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.806, 2024-02-22T02:38:35.101 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.806 2024-02-22T02:38:35.101 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.101 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.101 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.554, 2024-02-22T02:38:35.101 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.806, 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.806 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.371 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:34 2024", 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.463, 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.539, 2024-02-22T02:38:35.102 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.493 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.610, 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.915, 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.915 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.610, 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.915, 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.915 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.344 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-02-22T02:38:35.103 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:38:35.110 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:35.110 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2024-02-22T02:38:35.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2024-02-22T02:38:35.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:35.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2024-02-22T02:38:35.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2024-02-22T02:38:35.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:38:35.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2024-02-22T02:38:35.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-02-22T02:38:35.186 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-02-22T02:38:35.186 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:38:35.186 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:38:35.187 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:38:35.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-mgr.x.asok 2024-02-22T02:38:35.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:38:35.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-mgr.x.asok dump_osd_network 0 2024-02-22T02:38:35.287 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:38:35.288 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 0, 2024-02-22T02:38:35.288 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [ 2024-02-22T02:38:35.288 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.288 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:28 2024", 2024-02-22T02:38:35.288 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.288 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 1, 2024-02-22T02:38:35.288 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:38:35.288 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:35.288 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.288 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.603, 2024-02-22T02:38:35.288 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.620, 2024-02-22T02:38:35.288 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.660 2024-02-22T02:38:35.288 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.289 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.289 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.555, 2024-02-22T02:38:35.289 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.533, 2024-02-22T02:38:35.289 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.533 2024-02-22T02:38:35.289 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.289 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.289 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.655, 2024-02-22T02:38:35.289 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.738, 2024-02-22T02:38:35.289 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.738 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.488 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:27 2024", 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 2, 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.559, 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.605, 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.624 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.290 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.488, 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.488, 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.488 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.631, 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.753, 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.753 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.268 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:28 2024", 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.613, 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.621, 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.604 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.537, 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.432, 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.432 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.675, 2024-02-22T02:38:35.292 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.774, 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.774 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.481 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:27 2024", 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 2, 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 0, 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.518, 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.547, 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.586 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.293 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.294 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.485, 2024-02-22T02:38:35.294 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.445, 2024-02-22T02:38:35.294 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.445 2024-02-22T02:38:35.294 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.294 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.294 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.562, 2024-02-22T02:38:35.294 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.710, 2024-02-22T02:38:35.294 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.710 2024-02-22T02:38:35.294 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.299 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:27 2024", 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 2, 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.583, 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.570, 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.574 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.295 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.522, 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.462, 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.462 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.643, 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.665, 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.665 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.345 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:28 2024", 2024-02-22T02:38:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.583, 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.574, 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.536 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.508, 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.464, 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.464 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.297 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.671, 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.806, 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.806 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.554 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:28 2024", 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 1, 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.521, 2024-02-22T02:38:35.298 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.576, 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.582 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.459, 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.459, 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.459 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.604, 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.738, 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.738 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.420 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.299 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.300 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:29 2024", 2024-02-22T02:38:35.300 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.300 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:35.300 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:38:35.300 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:35.300 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.300 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.560, 2024-02-22T02:38:35.300 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.581, 2024-02-22T02:38:35.300 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.493 2024-02-22T02:38:35.300 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.476, 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.370, 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.251 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.649, 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 1.380, 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 1.380 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.434 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.301 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:30 2024", 2024-02-22T02:38:35.302 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.302 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 1, 2024-02-22T02:38:35.302 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 0, 2024-02-22T02:38:35.302 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:35.302 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.302 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.574, 2024-02-22T02:38:35.302 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.559, 2024-02-22T02:38:35.302 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.545 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.520, 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.460, 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.287 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.645, 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.687, 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.687 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.608 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:29 2024", 2024-02-22T02:38:35.303 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.304 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:35.304 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:38:35.304 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:35.304 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.304 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.514, 2024-02-22T02:38:35.304 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.558, 2024-02-22T02:38:35.305 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.493 2024-02-22T02:38:35.305 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.305 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.305 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.434, 2024-02-22T02:38:35.305 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.340, 2024-02-22T02:38:35.305 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.227 2024-02-22T02:38:35.305 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.305 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.305 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.564, 2024-02-22T02:38:35.305 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.915, 2024-02-22T02:38:35.305 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.915 2024-02-22T02:38:35.305 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.610 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:30 2024", 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 1, 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 0, 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.537, 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.498, 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.498 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.460, 2024-02-22T02:38:35.306 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.392, 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.266 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.646, 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.779, 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.779 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.551 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:27 2024", 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 2, 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 0, 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:35.307 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.440, 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.502, 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.512 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.426, 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.426, 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.426 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.459, 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.677, 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.677 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.326 2024-02-22T02:38:35.309 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-02-22T02:38:35.309 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-02-22T02:38:35.309 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:38:35.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:35.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2024-02-22T02:38:35.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2024-02-22T02:38:35.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2024-02-22T02:38:35.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:35.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2024-02-22T02:38:35.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2024-02-22T02:38:55.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2024-02-22T02:38:55.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-22T02:38:55.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-22T02:38:55.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-22T02:38:55.843 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-02-22T02:38:55.843 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-02-22T02:38:55.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-22T02:38:55.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:38:55.843 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-22T02:38:56.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-02-22T02:38:56.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-02-22T02:38:56.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-02-22T02:38:56.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:38:56.010 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-22T02:38:56.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-02-22T02:38:56.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-02-22T02:38:56.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672984' 2024-02-22T02:38:56.158 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:38:56.158 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-22T02:38:56.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-02-22T02:38:56.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-02-22T02:38:56.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672984 2-64424509462' 2024-02-22T02:38:56.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:38:56.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-02-22T02:38:56.311 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:38:56.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-22T02:38:56.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-02-22T02:38:56.312 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:38:56.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-02-22T02:38:56.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-02-22T02:38:56.314 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836514 2024-02-22T02:38:56.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-22T02:38:56.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836514 2024-02-22T02:38:56.731 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:38:56.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-02-22T02:38:56.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:38:56.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-22T02:38:56.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-02-22T02:38:56.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:38:56.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-02-22T02:38:56.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-02-22T02:38:56.736 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672984 2024-02-22T02:38:56.736 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-22T02:38:57.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-02-22T02:38:57.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:38:57.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-02-22T02:38:57.162 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:38:57.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-22T02:38:57.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-02-22T02:38:57.165 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:38:57.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-02-22T02:38:57.167 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509462 2024-02-22T02:38:57.167 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-02-22T02:38:57.167 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-22T02:38:57.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-02-22T02:38:57.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:38:57.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2024-02-22T02:38:57.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:38:57.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:38:57.582 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:38:57.582 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:38:57.582 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:38:57.582 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-osd.0.asok 2024-02-22T02:38:57.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:38:57.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-osd.0.asok dump_osd_network 2024-02-22T02:38:57.686 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:38:57.686 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 1000, 2024-02-22T02:38:57.686 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [] 2024-02-22T02:38:57.687 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:38:57.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:57.700 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2024-02-22T02:38:57.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2024-02-22T02:38:57.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:57.731 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2024-02-22T02:38:57.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2024-02-22T02:38:57.761 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:38:57.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2024-02-22T02:38:57.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-02-22T02:38:57.762 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-02-22T02:38:57.763 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:38:57.763 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:38:57.763 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:38:57.763 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-mgr.x.asok 2024-02-22T02:38:57.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:38:57.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-mgr.x.asok dump_osd_network 2024-02-22T02:38:57.860 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:38:57.861 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 1000, 2024-02-22T02:38:57.861 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [] 2024-02-22T02:38:57.861 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:38:57.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:57.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2024-02-22T02:38:57.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2024-02-22T02:38:57.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:57.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2024-02-22T02:38:57.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2024-02-22T02:38:57.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:38:57.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2024-02-22T02:38:57.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:38:57.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:38:57.952 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:38:57.952 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:38:57.952 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:38:57.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-osd.0.asok 2024-02-22T02:38:57.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:38:57.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-osd.0.asok dump_osd_network 0 2024-02-22T02:38:58.067 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:38:58.067 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 0, 2024-02-22T02:38:58.067 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [ 2024-02-22T02:38:58.067 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.068 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:53 2024", 2024-02-22T02:38:58.068 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.068 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:58.068 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:38:58.068 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:58.068 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.068 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.615, 2024-02-22T02:38:58.068 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.569, 2024-02-22T02:38:58.068 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.595 2024-02-22T02:38:58.068 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.068 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.068 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.692, 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.725, 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.774 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.692, 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.725, 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.774 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.622 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:54 2024", 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:58.069 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.569, 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.538, 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.517 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.603, 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.660, 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.915 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.603, 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.660, 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.915 2024-02-22T02:38:58.070 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.547 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:53 2024", 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.539, 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.538, 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.543 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.556, 2024-02-22T02:38:58.071 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.718, 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.806 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.556, 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.718, 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.806 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.476 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:54 2024", 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:38:58.072 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.485, 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.496, 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.512 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.612, 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.643, 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 1.380 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.612, 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.643, 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 1.380 2024-02-22T02:38:58.073 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.074 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.580 2024-02-22T02:38:58.074 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-02-22T02:38:58.074 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-02-22T02:38:58.074 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:38:58.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:58.082 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2024-02-22T02:38:58.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2024-02-22T02:38:58.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:58.121 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2024-02-22T02:38:58.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2024-02-22T02:38:58.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:38:58.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2024-02-22T02:38:58.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-02-22T02:38:58.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-02-22T02:38:58.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:38:58.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:38:58.154 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:38:58.154 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-mgr.x.asok 2024-02-22T02:38:58.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:38:58.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-mgr.x.asok dump_osd_network 0 2024-02-22T02:38:58.256 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:38:58.256 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 0, 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [ 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:52 2024", 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 2, 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.642, 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.600, 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.613 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.257 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.575, 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.268, 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.268 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.704, 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.734, 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.753 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.605 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:53 2024", 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 1, 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:38:58.258 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.580, 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.588, 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.629 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.497, 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.422, 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.422 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.657, 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.721, 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.738 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.259 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.589 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:53 2024", 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.615, 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.569, 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.595 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.506, 2024-02-22T02:38:58.260 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.323, 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.323 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.692, 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.725, 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.774 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.651 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:52 2024", 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 2, 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:38:58.261 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.580, 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.555, 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.567 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.501, 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.294, 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.294 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.659, 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.691, 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.691 2024-02-22T02:38:58.262 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.566 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:54 2024", 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.569, 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.538, 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.517 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.263 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.515, 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.344, 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.227 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.603, 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.660, 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.915 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.502 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:52 2024", 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 2, 2024-02-22T02:38:58.264 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 0, 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.566, 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.552, 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.568 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.470, 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.299, 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.299 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.611, 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.709, 2024-02-22T02:38:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.710 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.480 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:53 2024", 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 1, 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.551, 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.543, 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.568 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.266 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.469, 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.420, 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.420 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.631, 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.641, 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.738 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.633 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:55 2024", 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 1, 2024-02-22T02:38:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 0, 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.523, 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.565, 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.550 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.432, 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.432, 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.287 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.619, 2024-02-22T02:38:58.268 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.705, 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.705 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.441 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:53 2024", 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.539, 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.538, 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.543 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.269 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.519, 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.359, 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.359 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.556, 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.718, 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.806 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.705 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:52 2024", 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.270 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 2, 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 0, 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.540, 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.499, 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.506 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.439, 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.326, 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.326 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.657, 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.657, 2024-02-22T02:38:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.677 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.403 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:54 2024", 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.485, 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.496, 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.512 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.272 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.440, 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.309, 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.251 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.612, 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.643, 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 1.380 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.577 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:38:55 2024", 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 1, 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 0, 2024-02-22T02:38:58.273 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.502, 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.507, 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.499 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.436, 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.378, 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.359 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.577, 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.597, 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.779 2024-02-22T02:38:58.274 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:38:58.275 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.517 2024-02-22T02:38:58.275 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-02-22T02:38:58.275 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-02-22T02:38:58.275 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:38:58.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:58.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2024-02-22T02:38:58.308 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2024-02-22T02:38:58.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:38:58.309 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2024-02-22T02:38:58.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2024-02-22T02:38:58.349 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2024-02-22T02:39:48.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2024-02-22T02:39:48.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-22T02:39:48.353 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-22T02:39:48.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-22T02:39:48.816 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-02-22T02:39:48.816 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-02-22T02:39:48.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-22T02:39:48.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:39:48.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-22T02:39:49.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836527 2024-02-22T02:39:49.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836527 2024-02-22T02:39:49.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527' 2024-02-22T02:39:49.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:39:49.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-22T02:39:49.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672996 2024-02-22T02:39:49.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672996 2024-02-22T02:39:49.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-42949672996' 2024-02-22T02:39:49.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:39:49.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-22T02:39:49.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509475 2024-02-22T02:39:49.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509475 2024-02-22T02:39:49.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-42949672996 2-64424509475' 2024-02-22T02:39:49.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:39:49.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836527 2024-02-22T02:39:49.339 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:39:49.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-22T02:39:49.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836527 2024-02-22T02:39:49.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:39:49.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836527 2024-02-22T02:39:49.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-02-22T02:39:49.342 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836527 2024-02-22T02:39:49.342 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-22T02:39:49.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836527 2024-02-22T02:39:49.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-22T02:39:50.765 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-22T02:39:50.766 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-22T02:39:51.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836527 2024-02-22T02:39:51.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:39:51.246 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672996 2024-02-22T02:39:51.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:39:51.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-22T02:39:51.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672996 2024-02-22T02:39:51.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:39:51.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672996 2024-02-22T02:39:51.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-02-22T02:39:51.250 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672996 2024-02-22T02:39:51.250 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-22T02:39:51.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672997 -lt 42949672996 2024-02-22T02:39:51.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:39:51.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509475 2024-02-22T02:39:51.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:39:51.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-22T02:39:51.710 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:39:51.710 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509475 2024-02-22T02:39:51.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509475 2024-02-22T02:39:51.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-02-22T02:39:51.712 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509475 2024-02-22T02:39:51.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-22T02:39:52.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509475 -lt 64424509475 2024-02-22T02:39:52.132 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:39:52.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2024-02-22T02:39:52.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:39:52.133 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:39:52.134 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:39:52.134 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:52.134 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:52.134 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-osd.0.asok 2024-02-22T02:39:52.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:39:52.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-osd.0.asok dump_osd_network 2024-02-22T02:39:52.246 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:39:52.246 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 1000, 2024-02-22T02:39:52.246 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [] 2024-02-22T02:39:52.246 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:39:52.260 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:39:52.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2024-02-22T02:39:52.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2024-02-22T02:39:52.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:39:52.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2024-02-22T02:39:52.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2024-02-22T02:39:52.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:39:52.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2024-02-22T02:39:52.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-02-22T02:39:52.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-02-22T02:39:52.330 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:39:52.331 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:52.331 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:52.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-mgr.x.asok 2024-02-22T02:39:52.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:39:52.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-mgr.x.asok dump_osd_network 2024-02-22T02:39:52.430 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:39:52.430 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 1000, 2024-02-22T02:39:52.430 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [] 2024-02-22T02:39:52.430 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:39:52.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:39:52.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2024-02-22T02:39:52.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2024-02-22T02:39:52.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2024-02-22T02:39:52.474 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:39:52.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2024-02-22T02:39:52.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:39:52.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2024-02-22T02:39:52.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:39:52.512 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:39:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:39:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:52.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-osd.0.asok 2024-02-22T02:39:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:39:52.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-osd.0.asok dump_osd_network 0 2024-02-22T02:39:52.621 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:39:52.621 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 0, 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [ 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:48 2024", 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.593, 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.589, 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.595 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.631, 2024-02-22T02:39:52.622 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.720, 2024-02-22T02:39:52.623 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.725 2024-02-22T02:39:52.623 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.623 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.623 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.631, 2024-02-22T02:39:52.623 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.720, 2024-02-22T02:39:52.623 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.725 2024-02-22T02:39:52.623 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.623 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.505 2024-02-22T02:39:52.623 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.623 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.623 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:48 2024", 2024-02-22T02:39:52.623 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.623 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:39:52.624 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:39:52.624 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:39:52.624 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.624 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.530, 2024-02-22T02:39:52.624 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.541, 2024-02-22T02:39:52.624 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.549 2024-02-22T02:39:52.624 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.624 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.624 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.598, 2024-02-22T02:39:52.625 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.669, 2024-02-22T02:39:52.625 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.718 2024-02-22T02:39:52.625 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.625 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.625 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.598, 2024-02-22T02:39:52.625 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.669, 2024-02-22T02:39:52.625 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.718 2024-02-22T02:39:52.625 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.353 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:49 2024", 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.507, 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.542, 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.523 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.619, 2024-02-22T02:39:52.626 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.651, 2024-02-22T02:39:52.627 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.755 2024-02-22T02:39:52.627 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.627 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.627 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.619, 2024-02-22T02:39:52.627 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.651, 2024-02-22T02:39:52.627 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.755 2024-02-22T02:39:52.627 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.627 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.440 2024-02-22T02:39:52.627 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.627 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.627 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:49 2024", 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.528, 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.517, 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.539 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.753, 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.753, 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.753 2024-02-22T02:39:52.628 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.629 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.629 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.753, 2024-02-22T02:39:52.629 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.753, 2024-02-22T02:39:52.629 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.753 2024-02-22T02:39:52.629 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.629 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.418 2024-02-22T02:39:52.629 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-02-22T02:39:52.629 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-02-22T02:39:52.629 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:39:52.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:39:52.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2024-02-22T02:39:52.673 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2024-02-22T02:39:52.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:39:52.674 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2024-02-22T02:39:52.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2024-02-22T02:39:52.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:39:52.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2024-02-22T02:39:52.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-02-22T02:39:52.716 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-02-22T02:39:52.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:39:52.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:52.717 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:52.717 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-mgr.x.asok 2024-02-22T02:39:52.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:39:52.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-mgr.x.asok dump_osd_network 0 2024-02-22T02:39:52.817 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:39:52.817 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 0, 2024-02-22T02:39:52.817 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [ 2024-02-22T02:39:52.817 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.817 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:48 2024", 2024-02-22T02:39:52.817 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.817 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 1, 2024-02-22T02:39:52.817 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:39:52.817 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:39:52.817 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.817 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.560, 2024-02-22T02:39:52.817 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.597, 2024-02-22T02:39:52.818 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.594 2024-02-22T02:39:52.818 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.818 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.818 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.508, 2024-02-22T02:39:52.818 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.478, 2024-02-22T02:39:52.818 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.351 2024-02-22T02:39:52.818 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.818 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.818 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.615, 2024-02-22T02:39:52.818 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.730, 2024-02-22T02:39:52.818 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.730 2024-02-22T02:39:52.818 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.641 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:48 2024", 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.593, 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.589, 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.595 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.819 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.557, 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.510, 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.327 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.631, 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.720, 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.725 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.625 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:47 2024", 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.820 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 2, 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.582, 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.585, 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.593 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.511, 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.452, 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.328 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.821 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.658, 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.706, 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.734 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.405 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:44 2024", 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.581, 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.558, 2024-02-22T02:39:52.822 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.519 2024-02-22T02:39:52.823 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.823 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.823 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.542, 2024-02-22T02:39:52.823 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.438, 2024-02-22T02:39:52.823 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.254 2024-02-22T02:39:52.823 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.823 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.823 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.626, 2024-02-22T02:39:52.823 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.755, 2024-02-22T02:39:52.823 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.755 2024-02-22T02:39:52.823 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.823 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.590 2024-02-22T02:39:52.823 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.824 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.824 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:48 2024", 2024-02-22T02:39:52.824 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.824 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 1, 2024-02-22T02:39:52.824 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:39:52.824 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:39:52.824 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.824 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.553, 2024-02-22T02:39:52.824 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.570, 2024-02-22T02:39:52.824 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.551 2024-02-22T02:39:52.824 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.824 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.824 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.472, 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.472, 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.418 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.642, 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.728, 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.728 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.578 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:47 2024", 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 2, 2024-02-22T02:39:52.825 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 0, 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.461, 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.541, 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.557 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.430, 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.430, 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.263 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.555, 2024-02-22T02:39:52.826 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.648, 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.773 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.378 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:47 2024", 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 2, 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 0, 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.557, 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.556, 2024-02-22T02:39:52.827 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.523 2024-02-22T02:39:52.828 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.828 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.828 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.519, 2024-02-22T02:39:52.828 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.418, 2024-02-22T02:39:52.828 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.347 2024-02-22T02:39:52.828 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.828 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.828 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.607, 2024-02-22T02:39:52.828 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.777, 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.883 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.358 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:47 2024", 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 2, 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.483, 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.529, 2024-02-22T02:39:52.829 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.556 2024-02-22T02:39:52.830 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.830 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.830 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.467, 2024-02-22T02:39:52.830 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.430, 2024-02-22T02:39:52.830 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.294 2024-02-22T02:39:52.830 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.830 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.830 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.508, 2024-02-22T02:39:52.830 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.653, 2024-02-22T02:39:52.830 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.821 2024-02-22T02:39:52.830 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.830 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.328 2024-02-22T02:39:52.830 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:48 2024", 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 2, 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.530, 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.541, 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.549 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.494, 2024-02-22T02:39:52.831 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.430, 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.289 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.598, 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.669, 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.718 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.494 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:45 2024", 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 1, 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 0, 2024-02-22T02:39:52.832 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:39:52.833 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.833 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.496, 2024-02-22T02:39:52.833 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.541, 2024-02-22T02:39:52.833 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.546 2024-02-22T02:39:52.833 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.833 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.833 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.434, 2024-02-22T02:39:52.833 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.434, 2024-02-22T02:39:52.833 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.279 2024-02-22T02:39:52.833 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.833 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.833 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.573, 2024-02-22T02:39:52.833 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.683, 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.705 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.534 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:45 2024", 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 1, 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 0, 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "back", 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.545, 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.517, 2024-02-22T02:39:52.834 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.516 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.488, 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.431, 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.315 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.637, 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.637, 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.664 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.727 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: { 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: "last update": "Thu Feb 22 02:39:44 2024", 2024-02-22T02:39:52.835 INFO:tasks.workunit.client.0.smithi046.stdout: "stale": false, 2024-02-22T02:39:52.836 INFO:tasks.workunit.client.0.smithi046.stdout: "from osd": 0, 2024-02-22T02:39:52.836 INFO:tasks.workunit.client.0.smithi046.stdout: "to osd": 1, 2024-02-22T02:39:52.836 INFO:tasks.workunit.client.0.smithi046.stdout: "interface": "front", 2024-02-22T02:39:52.836 INFO:tasks.workunit.client.0.smithi046.stdout: "average": { 2024-02-22T02:39:52.836 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.504, 2024-02-22T02:39:52.836 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.518, 2024-02-22T02:39:52.836 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.535 2024-02-22T02:39:52.836 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.836 INFO:tasks.workunit.client.0.smithi046.stdout: "min": { 2024-02-22T02:39:52.836 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.452, 2024-02-22T02:39:52.836 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.445, 2024-02-22T02:39:52.836 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.344 2024-02-22T02:39:52.837 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.837 INFO:tasks.workunit.client.0.smithi046.stdout: "max": { 2024-02-22T02:39:52.837 INFO:tasks.workunit.client.0.smithi046.stdout: "1min": 0.590, 2024-02-22T02:39:52.837 INFO:tasks.workunit.client.0.smithi046.stdout: "5min": 0.622, 2024-02-22T02:39:52.837 INFO:tasks.workunit.client.0.smithi046.stdout: "15min": 0.729 2024-02-22T02:39:52.837 INFO:tasks.workunit.client.0.smithi046.stdout: }, 2024-02-22T02:39:52.837 INFO:tasks.workunit.client.0.smithi046.stdout: "last": 0.539 2024-02-22T02:39:52.837 INFO:tasks.workunit.client.0.smithi046.stdout: } 2024-02-22T02:39:52.837 INFO:tasks.workunit.client.0.smithi046.stdout: ] 2024-02-22T02:39:52.837 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:39:52.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:39:52.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2024-02-22T02:39:52.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2024-02-22T02:39:52.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:39:52.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2024-02-22T02:39:52.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2024-02-22T02:39:52.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:39:52.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2024-02-22T02:39:52.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-02-22T02:39:52.903 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-02-22T02:39:52.903 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:39:52.903 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:52.904 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:52.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-mgr.x.asok 2024-02-22T02:39:52.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:39:52.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-mgr.x.asok dump_osd_network 99 2024-02-22T02:39:53.008 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:39:53.008 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 99, 2024-02-22T02:39:53.008 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [] 2024-02-22T02:39:53.008 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:39:53.022 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:39:53.023 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2024-02-22T02:39:53.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2024-02-22T02:39:53.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2024-02-22T02:39:53.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2024-02-22T02:39:53.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:39:53.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:39:53.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:39:53.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:53.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-osd.0.asok 2024-02-22T02:39:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2024-02-22T02:39:53.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34135/ceph-osd.0.asok dump_osd_network 98 2024-02-22T02:39:53.171 INFO:tasks.workunit.client.0.smithi046.stdout:{ 2024-02-22T02:39:53.171 INFO:tasks.workunit.client.0.smithi046.stdout: "threshold": 98, 2024-02-22T02:39:53.171 INFO:tasks.workunit.client.0.smithi046.stdout: "entries": [] 2024-02-22T02:39:53.171 INFO:tasks.workunit.client.0.smithi046.stdout:} 2024-02-22T02:39:53.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2024-02-22T02:39:53.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2024-02-22T02:39:53.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2024-02-22T02:39:53.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2024-02-22T02:39:53.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-02-22T02:39:53.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-02-22T02:39:53.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:39:53.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-02-22T02:39:53.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:39:53.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:39:53.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:39:53.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:39:53.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:39:53.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:39:53.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:39:53.540 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:39:53.541 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:39:53.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:39:53.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:39:53.544 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:39:53.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:39:53.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:39:53.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:39:53.546 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:39:53.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:39:53.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:39:53.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-02-22T02:39:53.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:39:53.577 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:53.578 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:53.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34135 2024-02-22T02:39:53.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:39:53.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:39:53.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-02-22T02:39:53.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-02-22T02:39:53.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-02-22T02:39:53.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-02-22T02:39:53.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-02-22T02:39:53.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:39:53.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-02-22T02:39:53.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:39:53.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:39:53.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:39:53.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:39:53.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:39:53.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:39:53.584 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:39:53.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:39:53.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:39:53.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:39:53.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:39:53.587 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:39:53.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:39:53.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:39:53.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:39:53.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:39:53.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:39:53.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:39:53.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-02-22T02:39:53.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:39:53.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:53.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:53.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34135 2024-02-22T02:39:53.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:39:53.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:39:53.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-02-22T02:39:53.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:39:53.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:53.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:53.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.34135 2024-02-22T02:39:53.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:39:53.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:39:53.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:39:53.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-02-22T02:39:53.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2024-02-22T02:39:53.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2024-02-22T02:39:53.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2024-02-22T02:39:53.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2024-02-22T02:39:53.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2024-02-22T02:39:53.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2024-02-22T02:39:53.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-02-22T02:39:53.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:39:53.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:39:53.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:39:53.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-02-22T02:39:53.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-02-22T02:39:54.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:39:54.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:39:54.005 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:39:54.005 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:39:54.005 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:54.006 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:54.006 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:39:54.006 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-22T02:39:54.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:39:54.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:39:54.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:39:54.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:39:54.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:39:54.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:39:54.219 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:39:54.219 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:39:54.219 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:39:54.219 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:39:54.220 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:54.220 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:54.220 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-mon.a.asok 2024-02-22T02:39:54.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:39:54.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.34135/ceph-mon.a.asok config get fsid 2024-02-22T02:39:54.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:39:54.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:39:54.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:39:54.340 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:39:54.341 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:39:54.341 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:39:54.341 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:39:54.342 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:39:54.342 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:39:54.342 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:54.342 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:54.342 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34135/ceph-mon.a.asok 2024-02-22T02:39:54.342 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:39:54.342 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.34135/ceph-mon.a.asok config get mon_host 2024-02-22T02:39:54.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2024-02-22T02:39:54.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-02-22T02:39:54.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:39:54.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:39:54.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:39:54.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-02-22T02:39:54.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:39:55.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:39:55.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:39:55.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:39:55.390 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:39:55.390 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:55.390 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:55.390 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:39:55.390 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:39:55.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:39:55.430 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-02-22T02:39:55.431 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:39:55.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:39:55.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:39:55.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:39:55.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:39:55.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:39:55.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-02-22T02:39:55.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:39:55.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=612928a2-22f4-4d75-a3b2-121a2133f43d 2024-02-22T02:39:55.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 612928a2-22f4-4d75-a3b2-121a2133f43d' 2024-02-22T02:39:55.434 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 612928a2-22f4-4d75-a3b2-121a2133f43d 2024-02-22T02:39:55.434 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:39:55.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7s9Zl/Kv/GhAAm53jH5dhXAZktJSxUizH5Q== 2024-02-22T02:39:55.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7s9Zl/Kv/GhAAm53jH5dhXAZktJSxUizH5Q=="}' 2024-02-22T02:39:55.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 612928a2-22f4-4d75-a3b2-121a2133f43d -i td/network-ping/0/new.json 2024-02-22T02:39:55.745 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:39:55.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-02-22T02:39:55.763 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7s9Zl/Kv/GhAAm53jH5dhXAZktJSxUizH5Q== --osd-uuid 612928a2-22f4-4d75-a3b2-121a2133f43d 2024-02-22T02:39:55.800 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:39:55.800+0000 7f6d2853b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:39:55.813 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:39:55.812+0000 7f6d2853b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:39:55.814 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:39:55.816+0000 7f6d2853b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:39:55.815 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:39:55.816+0000 7f6d2853b7c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-02-22T02:39:55.815 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:39:55.816+0000 7f6d2853b7c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-02-22T02:39:55.815 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:39:55.816+0000 7f6d2853b7c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-02-22T02:39:55.853 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:39:55.852+0000 7f6d2853b7c0 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-02-22T02:40:02.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-02-22T02:40:02.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:40:02.365 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:40:02.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:40:02.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:40:02.945 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:40:02.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:40:02.945 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:40:02.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:40:02.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:40:02.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:40:02.980 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:02.980+0000 7f9c558e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:40:02.983 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:02.984+0000 7f9c558e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:40:02.984 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:02.984+0000 7f9c558e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:40:03.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:40:03.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:40:03.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:40:03.346 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:40:03.346 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:40:03.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:40:03.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:03.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:40:03.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:03.347 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:03.551 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:03.552+0000 7f9c558e37c0 -1 Falling back to public interface 2024-02-22T02:40:03.798 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:04.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:04.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:04.801 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:40:04.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:40:04.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:04.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:05.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:06.185 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:06.184+0000 7f9c558e37c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:40:06.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:06.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:06.220 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:40:06.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:40:06.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:06.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:06.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:07.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:07.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:07.868 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:40:07.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:40:07.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:07.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:08.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:09.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:09.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:09.311 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:40:09.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:40:09.311 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:09.312 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:09.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:10.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:10.792 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:10.792 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:40:10.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:40:10.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:10.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:11.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:12.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:12.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:12.345 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:40:12.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:40:12.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:12.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:12.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:13.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:13.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:13.784 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:40:13.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:40:13.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:13.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:14.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:15.373 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:15.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:15.374 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:40:15.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:40:15.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:15.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:15.799 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:16.800 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:16.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:16.801 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:40:16.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:40:16.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:16.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:17.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:18.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:18.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:18.386 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:40:18.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:40:18.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:18.387 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:18.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:19.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:19.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:19.816 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:40:19.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:40:19.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:19.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:20.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:21.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:21.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:21.417 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:40:21.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:40:21.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:21.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:22.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:23.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:23.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:23.216 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:40:23.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:40:23.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:23.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:24.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:25.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:25.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:25.052 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:40:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:40:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:25.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:25.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:26.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:26.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:26.907 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:40:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:40:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:26.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:27.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:28.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:28.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:28.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:40:28.356 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:40:28.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:28.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:29.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:30.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:30.487 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:40:30.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:30.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:40:30.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:30.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:40:30.987 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2506262207,v1:127.0.0.1:6803/2506262207] [v2:127.0.0.1:6804/2506262207,v1:127.0.0.1:6805/2506262207] exists,up 612928a2-22f4-4d75-a3b2-121a2133f43d 2024-02-22T02:40:30.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-02-22T02:40:30.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-02-22T02:40:30.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:40:30.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-02-22T02:40:30.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-02-22T02:40:30.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:40:30.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:40:30.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:40:30.991 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:40:30.991 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:40:30.991 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:40:30.991 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:40:30.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:40:30.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:40:30.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:40:30.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:40:30.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-02-22T02:40:30.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-02-22T02:40:30.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:40:30.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:40:30.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:40:30.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:40:30.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:40:30.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:40:30.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-02-22T02:40:30.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:40:30.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5585daca-a032-4712-a5cc-71a6f59769d7 2024-02-22T02:40:30.994 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 5585daca-a032-4712-a5cc-71a6f59769d7 2024-02-22T02:40:30.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5585daca-a032-4712-a5cc-71a6f59769d7' 2024-02-22T02:40:30.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:40:31.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAftNZlEpBCARAAfuqKU4kW8PgMCYb9EeaY6g== 2024-02-22T02:40:31.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAftNZlEpBCARAAfuqKU4kW8PgMCYb9EeaY6g=="}' 2024-02-22T02:40:31.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5585daca-a032-4712-a5cc-71a6f59769d7 -i td/network-ping/1/new.json 2024-02-22T02:40:31.547 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:40:31.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-02-22T02:40:31.566 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAftNZlEpBCARAAfuqKU4kW8PgMCYb9EeaY6g== --osd-uuid 5585daca-a032-4712-a5cc-71a6f59769d7 2024-02-22T02:40:31.606 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:31.607+0000 7f20e3be27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:40:31.608 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:31.607+0000 7f20e3be27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:40:31.609 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:31.607+0000 7f20e3be27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:40:31.610 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:31.611+0000 7f20e3be27c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-02-22T02:40:31.610 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:31.611+0000 7f20e3be27c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-02-22T02:40:31.610 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:31.611+0000 7f20e3be27c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-02-22T02:40:31.653 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:31.651+0000 7f20e3be27c0 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-02-22T02:40:38.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-02-22T02:40:38.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:40:38.090 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-02-22T02:40:38.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-22T02:40:38.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:40:38.703 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-02-22T02:40:38.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-22T02:40:38.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:40:38.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:40:38.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:40:38.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:40:38.740 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:38.739+0000 7f9500ce37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:40:38.742 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:38.743+0000 7f9500ce37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:40:38.743 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:38.743+0000 7f9500ce37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:40:39.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-22T02:40:39.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:40:39.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-22T02:40:39.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:40:39.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:40:39.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:39.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:40:39.169 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:40:39.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:39.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:40:39.323 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:39.323+0000 7f9500ce37c0 -1 Falling back to public interface 2024-02-22T02:40:39.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:40.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:40.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:40.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:40:40.604 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:40:40.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:40.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:40:41.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:41.985 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:40:41.983+0000 7f9500ce37c0 -1 osd.1 0 log_to_monitors true 2024-02-22T02:40:42.032 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:40:42.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:42.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:42.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:40:42.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:42.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:40:42.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:43.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:43.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:43.711 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:40:43.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:40:43.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:43.712 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:40:44.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:45.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:45.148 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:40:45.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:45.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:40:45.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:45.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:40:45.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:46.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:46.625 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:46.625 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:40:46.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:40:46.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:46.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:40:47.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:48.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:48.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:48.184 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:40:48.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:40:48.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:48.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:40:48.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:49.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:49.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:49.619 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:40:49.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:40:49.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:49.620 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:40:50.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:51.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:51.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:51.151 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:40:51.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:40:51.151 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:51.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:40:51.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:52.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:52.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:52.589 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:40:52.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:40:52.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:52.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:40:53.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:54.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:54.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:54.070 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:40:54.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:40:54.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:54.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:40:54.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:55.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:55.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:55.569 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:40:55.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:40:55.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:55.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:40:56.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:57.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:57.543 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:57.543 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:40:57.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:40:57.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:57.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:40:58.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:40:59.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:40:59.495 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:40:59.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:40:59.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:40:59.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:40:59.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:41:00.348 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:01.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:01.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:01.350 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:41:01.350 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:41:01.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:01.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:41:02.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:03.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:03.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:03.294 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:41:03.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:41:03.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:03.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:41:04.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:05.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:05.338 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:41:05.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:05.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:41:05.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:05.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:41:06.145 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:07.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:07.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:07.147 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:41:07.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:41:07.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:07.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:41:07.575 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3545014877,v1:127.0.0.1:6811/3545014877] [v2:127.0.0.1:6812/3545014877,v1:127.0.0.1:6813/3545014877] exists,up 5585daca-a032-4712-a5cc-71a6f59769d7 2024-02-22T02:41:07.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:41:07.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:41:07.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:41:07.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2024-02-22T02:41:07.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-02-22T02:41:07.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:41:07.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-22T02:41:07.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:41:07.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-02-22T02:41:07.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:41:07.579 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:41:07.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' 2024-02-22T02:41:07.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:41:07.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:41:07.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:41:07.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-02-22T02:41:07.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-02-22T02:41:07.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:41:07.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:41:07.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:41:07.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:41:07.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:41:07.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:41:07.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-02-22T02:41:07.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:41:07.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=542b551f-61b5-4587-a18e-2e77b8cd7bc6 2024-02-22T02:41:07.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 542b551f-61b5-4587-a18e-2e77b8cd7bc6' 2024-02-22T02:41:07.581 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 542b551f-61b5-4587-a18e-2e77b8cd7bc6 2024-02-22T02:41:07.581 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:41:07.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDtNZl4lsuJBAAY2cCkVXaTYVotQDUp8Tvjg== 2024-02-22T02:41:07.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDtNZl4lsuJBAAY2cCkVXaTYVotQDUp8Tvjg=="}' 2024-02-22T02:41:07.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 542b551f-61b5-4587-a18e-2e77b8cd7bc6 -i td/network-ping/2/new.json 2024-02-22T02:41:08.192 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:41:08.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-02-22T02:41:08.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBDtNZl4lsuJBAAY2cCkVXaTYVotQDUp8Tvjg== --osd-uuid 542b551f-61b5-4587-a18e-2e77b8cd7bc6 2024-02-22T02:41:08.242 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:41:08.243+0000 7efc15dc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:41:08.326 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:41:08.327+0000 7efc15dc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:41:08.328 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:41:08.327+0000 7efc15dc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:41:08.328 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:41:08.327+0000 7efc15dc17c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-02-22T02:41:08.328 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:41:08.327+0000 7efc15dc17c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-02-22T02:41:08.328 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:41:08.327+0000 7efc15dc17c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-02-22T02:41:08.416 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:41:08.419+0000 7efc15dc17c0 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-02-22T02:41:14.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-02-22T02:41:14.895 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:41:14.897 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-02-22T02:41:14.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-22T02:41:14.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:41:15.459 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-22T02:41:15.459 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-02-22T02:41:15.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:41:15.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:41:15.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b7b01b62-f171-44e3-bb3f-0986be9b0763 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34135/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:41:15.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:41:15.495 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:41:15.495+0000 7fe39bd8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:41:15.497 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:41:15.499+0000 7fe39bd8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:41:15.498 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:41:15.499+0000 7fe39bd8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:41:15.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-22T02:41:15.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:41:15.921 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:41:15.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-22T02:41:15.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:41:15.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:41:15.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:15.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:41:15.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:15.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:16.062 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:41:16.063+0000 7fe39bd8d7c0 -1 Falling back to public interface 2024-02-22T02:41:16.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:17.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:17.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:17.359 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:41:17.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:41:17.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:17.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:17.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:18.755 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:41:18.755+0000 7fe39bd8d7c0 -1 osd.2 0 log_to_monitors true 2024-02-22T02:41:18.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:18.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:18.785 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:41:18.786 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:41:18.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:18.786 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:19.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:20.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:20.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:20.318 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:41:20.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:41:20.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:20.319 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:20.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:21.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:21.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:21.782 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:41:21.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:41:21.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:21.784 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:22.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:23.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:23.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:23.268 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:41:23.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:41:23.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:23.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:23.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:24.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:24.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:24.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:41:24.855 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:41:24.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:24.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:25.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:26.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:26.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:26.325 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:41:26.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:41:26.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:26.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:26.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:27.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:27.823 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:27.823 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:41:27.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:41:27.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:27.824 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:28.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:29.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:29.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:29.255 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:41:29.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:41:29.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:29.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:29.762 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:30.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:30.767 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:30.767 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:41:30.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:41:30.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:30.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:31.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:32.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:32.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:32.230 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:41:32.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:41:32.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:32.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:32.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:33.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:33.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:33.960 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:41:33.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:41:33.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:33.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:34.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:35.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:35.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:35.759 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:41:35.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:41:35.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:35.759 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:36.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:37.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:37.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:37.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:41:37.603 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:41:37.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:37.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:38.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:39.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:39.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:39.726 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:41:39.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:41:39.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:39.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:40.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:41.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:41.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:41.570 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:41:41.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:41:41.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:41.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:42.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:43.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:43.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:43.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:41:43.493 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:41:43.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:43.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:43.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:41:44.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:41:44.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:41:44.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-22T02:41:44.997 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-02-22T02:41:44.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:41:44.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:41:45.415 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/650342876,v1:127.0.0.1:6819/650342876] [v2:127.0.0.1:6820/650342876,v1:127.0.0.1:6821/650342876] exists,up 542b551f-61b5-4587-a18e-2e77b8cd7bc6 2024-02-22T02:41:45.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:41:45.415 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:41:45.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:41:45.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2024-02-22T02:41:50.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2024-02-22T02:41:51.131 INFO:tasks.workunit.client.0.smithi046.stderr:bucket 'dc1' already exists 2024-02-22T02:41:51.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2024-02-22T02:41:51.708 INFO:tasks.workunit.client.0.smithi046.stderr:bucket 'dc2' already exists 2024-02-22T02:41:51.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2024-02-22T02:41:52.314 INFO:tasks.workunit.client.0.smithi046.stderr:bucket 'dc3' already exists 2024-02-22T02:41:52.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2024-02-22T02:41:52.997 INFO:tasks.workunit.client.0.smithi046.stderr:bucket 'rack1' already exists 2024-02-22T02:41:53.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2024-02-22T02:41:53.683 INFO:tasks.workunit.client.0.smithi046.stderr:bucket 'rack2' already exists 2024-02-22T02:41:53.700 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2024-02-22T02:41:54.252 INFO:tasks.workunit.client.0.smithi046.stderr:bucket 'rack3' already exists 2024-02-22T02:41:54.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2024-02-22T02:41:54.905 INFO:tasks.workunit.client.0.smithi046.stderr:bucket 'host1' already exists 2024-02-22T02:41:54.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2024-02-22T02:41:55.566 INFO:tasks.workunit.client.0.smithi046.stderr:bucket 'host2' already exists 2024-02-22T02:41:55.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2024-02-22T02:41:56.118 INFO:tasks.workunit.client.0.smithi046.stderr:bucket 'host3' already exists 2024-02-22T02:41:56.134 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2024-02-22T02:41:56.747 INFO:tasks.workunit.client.0.smithi046.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2024-02-22T02:41:56.764 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2024-02-22T02:41:57.351 INFO:tasks.workunit.client.0.smithi046.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2024-02-22T02:41:57.368 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2024-02-22T02:41:57.870 INFO:tasks.workunit.client.0.smithi046.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2024-02-22T02:41:57.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2024-02-22T02:41:58.521 INFO:tasks.workunit.client.0.smithi046.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2024-02-22T02:41:58.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2024-02-22T02:41:59.098 INFO:tasks.workunit.client.0.smithi046.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2024-02-22T02:41:59.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2024-02-22T02:41:59.852 INFO:tasks.workunit.client.0.smithi046.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2024-02-22T02:41:59.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2024-02-22T02:42:00.513 INFO:tasks.workunit.client.0.smithi046.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2024-02-22T02:42:00.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2024-02-22T02:42:01.099 INFO:tasks.workunit.client.0.smithi046.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2024-02-22T02:42:01.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2024-02-22T02:42:01.726 INFO:tasks.workunit.client.0.smithi046.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2024-02-22T02:42:01.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2024-02-22T02:42:02.301 INFO:tasks.workunit.client.0.smithi046.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2024-02-22T02:42:02.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2024-02-22T02:42:02.990 INFO:tasks.workunit.client.0.smithi046.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2024-02-22T02:42:03.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2024-02-22T02:42:03.575 INFO:tasks.workunit.client.0.smithi046.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2024-02-22T02:42:03.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2024-02-22T02:42:04.146 INFO:tasks.workunit.client.0.smithi046.stderr:rule myrule already exists 2024-02-22T02:42:04.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2024-02-22T02:42:04.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 16 replicated myrule 2024-02-22T02:42:04.856 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'foo' already exists 2024-02-22T02:42:04.873 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-22T02:42:05.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-02-22T02:42:06.129 INFO:tasks.workunit.client.0.smithi046.stdout:hints = 1 2024-02-22T02:42:06.130 INFO:tasks.workunit.client.0.smithi046.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-02-22T02:42:06.130 INFO:tasks.workunit.client.0.smithi046.stdout:Object prefix: benchmark_data_smithi046_46473 2024-02-22T02:42:06.130 INFO:tasks.workunit.client.0.smithi046.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-22T02:42:06.130 INFO:tasks.workunit.client.0.smithi046.stdout: 0 0 0 0 0 0 - 0 2024-02-22T02:42:07.129 INFO:tasks.workunit.client.0.smithi046.stdout: 1 16 56 40 0.156233 0.15625 0.387782 0.3662 2024-02-22T02:42:08.130 INFO:tasks.workunit.client.0.smithi046.stdout: 2 16 107 91 0.177712 0.199219 0.260031 0.330627 2024-02-22T02:42:09.130 INFO:tasks.workunit.client.0.smithi046.stdout: 3 16 164 148 0.192686 0.222656 0.31392 0.313458 2024-02-22T02:42:10.130 INFO:tasks.workunit.client.0.smithi046.stdout: 4 16 220 204 0.199194 0.21875 0.248274 0.303224 2024-02-22T02:42:11.130 INFO:tasks.workunit.client.0.smithi046.stdout: 5 16 276 260 0.203098 0.21875 0.28822 0.301135 2024-02-22T02:42:12.130 INFO:tasks.workunit.client.0.smithi046.stdout: 6 16 332 316 0.205703 0.21875 0.301266 0.294503 2024-02-22T02:42:13.130 INFO:tasks.workunit.client.0.smithi046.stdout: 7 16 396 380 0.212026 0.25 0.24222 0.290641 2024-02-22T02:42:14.130 INFO:tasks.workunit.client.0.smithi046.stdout: 8 16 452 436 0.212862 0.21875 0.279667 0.289327 2024-02-22T02:42:15.131 INFO:tasks.workunit.client.0.smithi046.stdout: 9 16 508 492 0.213513 0.21875 0.256449 0.286139 2024-02-22T02:42:16.131 INFO:tasks.workunit.client.0.smithi046.stdout: 10 16 580 564 0.220284 0.28125 0.266579 0.279854 2024-02-22T02:42:17.131 INFO:tasks.workunit.client.0.smithi046.stdout:Total time run: 10.1558 2024-02-22T02:42:17.131 INFO:tasks.workunit.client.0.smithi046.stdout:Total writes made: 580 2024-02-22T02:42:17.131 INFO:tasks.workunit.client.0.smithi046.stdout:Write size: 4096 2024-02-22T02:42:17.131 INFO:tasks.workunit.client.0.smithi046.stdout:Object size: 4096 2024-02-22T02:42:17.131 INFO:tasks.workunit.client.0.smithi046.stdout:Bandwidth (MB/sec): 0.223086 2024-02-22T02:42:17.131 INFO:tasks.workunit.client.0.smithi046.stdout:Stddev Bandwidth: 0.031905 2024-02-22T02:42:17.131 INFO:tasks.workunit.client.0.smithi046.stdout:Max bandwidth (MB/sec): 0.28125 2024-02-22T02:42:17.131 INFO:tasks.workunit.client.0.smithi046.stdout:Min bandwidth (MB/sec): 0.15625 2024-02-22T02:42:17.131 INFO:tasks.workunit.client.0.smithi046.stdout:Average IOPS: 57 2024-02-22T02:42:17.132 INFO:tasks.workunit.client.0.smithi046.stdout:Stddev IOPS: 8.16769 2024-02-22T02:42:17.132 INFO:tasks.workunit.client.0.smithi046.stdout:Max IOPS: 72 2024-02-22T02:42:17.132 INFO:tasks.workunit.client.0.smithi046.stdout:Min IOPS: 40 2024-02-22T02:42:17.132 INFO:tasks.workunit.client.0.smithi046.stdout:Average Latency(s): 0.278763 2024-02-22T02:42:17.132 INFO:tasks.workunit.client.0.smithi046.stdout:Stddev Latency(s): 0.0472777 2024-02-22T02:42:17.132 INFO:tasks.workunit.client.0.smithi046.stdout:Max latency(s): 0.567858 2024-02-22T02:42:17.132 INFO:tasks.workunit.client.0.smithi046.stdout:Min latency(s): 0.208547 2024-02-22T02:42:17.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2024-02-22T02:42:27.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2024-02-22T02:42:27.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-22T02:42:27.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-22T02:42:27.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-22T02:42:27.705 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-02-22T02:42:27.705 INFO:tasks.workunit.client.0.smithi046.stderr:2' 2024-02-22T02:42:27.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-22T02:42:27.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:42:27.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-22T02:42:27.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-02-22T02:42:27.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-02-22T02:42:27.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-02-22T02:42:27.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:42:27.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-22T02:42:28.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-02-22T02:42:28.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-02-22T02:42:28.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672987' 2024-02-22T02:42:28.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:42:28.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-22T02:42:28.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-02-22T02:42:28.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-02-22T02:42:28.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672987 2-64424509455' 2024-02-22T02:42:28.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:42:28.188 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-02-22T02:42:28.188 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:42:28.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-22T02:42:28.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-02-22T02:42:28.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:42:28.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-02-22T02:42:28.191 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836508 2024-02-22T02:42:28.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-02-22T02:42:28.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-22T02:42:28.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-02-22T02:42:28.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:42:28.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-02-22T02:42:28.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:42:28.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-22T02:42:28.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-02-22T02:42:28.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:42:28.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-02-22T02:42:28.618 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672987 2024-02-22T02:42:28.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-02-22T02:42:28.619 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-22T02:42:29.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672987 2024-02-22T02:42:29.077 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:42:29.077 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-02-22T02:42:29.078 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:42:29.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-22T02:42:29.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-02-22T02:42:29.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:42:29.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-02-22T02:42:29.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-02-22T02:42:29.083 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509455 2024-02-22T02:42:29.083 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-22T02:42:29.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-02-22T02:42:29.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2024-02-22T02:42:29.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2024-02-22T02:42:30.016 INFO:tasks.workunit.client.0.smithi046.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.627ms); Slow OSD heartbeats on front (longest 0.654ms); 1 pool(s) do not have an application enabled 2024-02-22T02:42:30.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2024-02-22T02:42:30.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2024-02-22T02:42:30.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2024-02-22T02:42:30.587 INFO:tasks.workunit.client.0.smithi046.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.627ms); Slow OSD heartbeats on front (longest 0.654ms); 1 pool(s) do not have an application enabled 2024-02-22T02:42:30.587 INFO:tasks.workunit.client.0.smithi046.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.627ms) 2024-02-22T02:42:30.588 INFO:tasks.workunit.client.0.smithi046.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.627 msec 2024-02-22T02:42:30.588 INFO:tasks.workunit.client.0.smithi046.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.615 msec 2024-02-22T02:42:30.588 INFO:tasks.workunit.client.0.smithi046.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.605 msec 2024-02-22T02:42:30.588 INFO:tasks.workunit.client.0.smithi046.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.562 msec 2024-02-22T02:42:30.588 INFO:tasks.workunit.client.0.smithi046.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.535 msec 2024-02-22T02:42:30.588 INFO:tasks.workunit.client.0.smithi046.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.491 msec 2024-02-22T02:42:30.588 INFO:tasks.workunit.client.0.smithi046.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.654ms) 2024-02-22T02:42:30.588 INFO:tasks.workunit.client.0.smithi046.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.654 msec 2024-02-22T02:42:30.588 INFO:tasks.workunit.client.0.smithi046.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.616 msec 2024-02-22T02:42:30.588 INFO:tasks.workunit.client.0.smithi046.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.608 msec 2024-02-22T02:42:30.588 INFO:tasks.workunit.client.0.smithi046.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.586 msec 2024-02-22T02:42:30.588 INFO:tasks.workunit.client.0.smithi046.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.560 msec 2024-02-22T02:42:30.588 INFO:tasks.workunit.client.0.smithi046.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.558 msec 2024-02-22T02:42:30.589 INFO:tasks.workunit.client.0.smithi046.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-22T02:42:30.589 INFO:tasks.workunit.client.0.smithi046.stdout: application not enabled on pool 'foo' 2024-02-22T02:42:30.589 INFO:tasks.workunit.client.0.smithi046.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-22T02:42:30.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2024-02-22T02:42:30.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2024-02-22T02:42:30.605 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2024-02-22T02:42:30.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2024-02-22T02:42:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-02-22T02:42:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-02-22T02:42:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:42:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-02-22T02:42:30.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:42:30.609 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:42:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:42:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:42:30.609 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:42:30.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:42:30.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:42:30.920 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:42:30.921 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:42:30.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:42:30.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:42:30.923 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:42:30.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:42:30.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:42:30.925 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:42:30.925 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:42:30.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:42:30.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:42:30.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-02-22T02:42:30.953 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:42:30.953 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:42:30.953 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:42:30.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34135 2024-02-22T02:42:30.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:42:30.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:42:30.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-22T02:42:30.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/network-ping 0 2024-02-22T02:42:30.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-02-22T02:42:30.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-22T02:42:30.955 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-02-22T02:42:30.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:42:30.955 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:42:30.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:42:30.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:42:30.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:42:30.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:42:30.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:42:30.959 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:42:30.959 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:42:30.960 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:42:30.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:42:30.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:42:30.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:42:30.961 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:42:30.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:42:30.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:42:30.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:42:30.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-22T02:42:30.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-02-22T02:42:30.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:42:30.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:42:30.966 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34135 2024-02-22T02:42:30.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34135 2024-02-22T02:42:30.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:42:30.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:42:30.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-22T02:42:30.968 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-22T02:42:30.968 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-22T02:42:30.981 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2024-02-22T02:42:30.981 DEBUG:teuthology.orchestra.run.smithi046:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbe7fcec16eddee6239e4dd68c4d0203c7df6461 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2024-02-22T02:42:31.037 INFO:tasks.workunit.client.0.smithi046.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-22T02:42:31.043 INFO:tasks.workunit.client.0.smithi046.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-22T02:42:31.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-22T02:42:31.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-22T02:42:31.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-22T02:42:31.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-22T02:42:31.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-22T02:42:31.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-22T02:42:31.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-22T02:42:31.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-22T02:42:31.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/ok-to-stop 2024-02-22T02:42:31.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2024-02-22T02:42:31.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2024-02-22T02:42:31.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2024-02-22T02:42:31.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2024-02-22T02:42:31.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2024-02-22T02:42:31.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2024-02-22T02:42:31.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2024-02-22T02:42:31.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2024-02-22T02:42:31.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2024-02-22T02:42:31.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2024-02-22T02:42:31.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2024-02-22T02:42:31.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2024-02-22T02:42:31.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2024-02-22T02:42:31.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2024-02-22T02:42:31.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none ' 2024-02-22T02:42:31.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none ' 2024-02-22T02:42:31.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none ' 2024-02-22T02:42:31.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2024-02-22T02:42:31.046 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-22T02:42:31.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2024-02-22T02:42:31.048 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_0_osd 2024-02-22T02:42:31.048 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_1_mon_checks 2024-02-22T02:42:31.048 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_2_mons_checks 2024-02-22T02:42:31.048 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_3_mons_checks 2024-02-22T02:42:31.048 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_4_mons_checks 2024-02-22T02:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr:TEST_5_mons_checks' 2024-02-22T02:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-02-22T02:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-02-22T02:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-02-22T02:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-02-22T02:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:42:31.049 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:42:31.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:42:31.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:42:31.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:42:31.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:42:31.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:42:31.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:42:31.053 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:42:31.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:42:31.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:42:31.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:42:31.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:42:31.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:42:31.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:42:31.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:42:31.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:42:31.058 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:42:31.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:42:31.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:42:31.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:42:31.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:42:31.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-02-22T02:42:31.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:42:31.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:42:31.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:42:31.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.46785 2024-02-22T02:42:31.064 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:42:31.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-22T02:42:31.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-22T02:42:31.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:42:31.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-02-22T02:42:31.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2024-02-22T02:42:31.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2024-02-22T02:42:31.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-02-22T02:42:31.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-02-22T02:42:31.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:42:31.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:42:31.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:42:31.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:42:31.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-02-22T02:42:31.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-02-22T02:42:31.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:42:31.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:42:31.459 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:42:31.460 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:42:31.460 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:42:31.460 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:42:31.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:42:31.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-02-22T02:42:31.669 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:42:31.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:42:31.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:42:31.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:42:31.670 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:42:31.671 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:42:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:42:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:42:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:42:31.671 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:42:31.672 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:42:31.672 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:42:31.672 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:42:31.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:42:31.672 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get fsid 2024-02-22T02:42:31.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:42:31.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:42:31.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:42:31.793 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:42:31.794 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:42:31.794 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:42:31.794 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:42:31.794 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:42:31.795 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:42:31.795 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:42:31.795 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:42:31.795 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:42:31.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:42:31.795 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get mon_host 2024-02-22T02:42:31.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2024-02-22T02:42:31.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-02-22T02:42:31.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:42:31.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:42:31.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:42:31.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-02-22T02:42:31.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:42:32.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:42:32.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:42:32.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:42:32.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:42:32.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:42:32.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:42:32.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:42:32.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:42:32.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:42:32.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2024-02-22T02:42:32.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-02-22T02:42:32.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:42:32.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:42:32.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:42:32.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-02-22T02:42:32.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-02-22T02:42:32.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:42:32.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:42:32.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:42:32.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-02-22T02:42:32.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-02-22T02:42:32.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:42:32.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:42:32.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-02-22T02:42:32.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:42:32.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:42:32.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:42:32.913 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:42:32.913 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:42:32.913 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:42:32.913 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:42:32.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:42:32.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:42:32.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:42:32.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:42:32.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-02-22T02:42:32.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-02-22T02:42:32.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:42:32.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:42:32.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:42:32.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:42:32.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:42:32.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:42:32.914 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-02-22T02:42:32.915 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:42:32.915 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d33fd331-a230-4b5d-b65f-84908a508883 2024-02-22T02:42:32.915 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 d33fd331-a230-4b5d-b65f-84908a508883 2024-02-22T02:42:32.916 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d33fd331-a230-4b5d-b65f-84908a508883' 2024-02-22T02:42:32.916 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:42:32.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCYtNZlWNOyNxAARdNzV6A9Tq/Mq3JhEGoUZw== 2024-02-22T02:42:32.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCYtNZlWNOyNxAARdNzV6A9Tq/Mq3JhEGoUZw=="}' 2024-02-22T02:42:32.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d33fd331-a230-4b5d-b65f-84908a508883 -i td/ok-to-stop/0/new.json 2024-02-22T02:42:33.232 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:42:33.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-02-22T02:42:33.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYtNZlWNOyNxAARdNzV6A9Tq/Mq3JhEGoUZw== --osd-uuid d33fd331-a230-4b5d-b65f-84908a508883 2024-02-22T02:42:33.273 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:42:33.274+0000 7f10d86427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:42:33.309 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:42:33.310+0000 7f10d86427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:42:33.310 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:42:33.310+0000 7f10d86427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:42:33.310 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:42:33.310+0000 7f10d86427c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-02-22T02:42:33.311 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:42:33.310+0000 7f10d86427c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-02-22T02:42:33.311 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:42:33.310+0000 7f10d86427c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-02-22T02:42:33.348 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:42:33.350+0000 7f10d86427c0 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-02-22T02:42:39.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-02-22T02:42:39.860 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:42:39.861 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:42:39.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:42:39.861 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:42:40.430 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:42:40.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:42:40.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:42:40.431 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:42:40.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:42:40.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:42:40.461 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:42:40.462+0000 7f7a7d4c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:42:40.490 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:42:40.490+0000 7f7a7d4c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:42:40.491 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:42:40.490+0000 7f7a7d4c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:42:40.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:42:40.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:42:40.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:42:40.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:42:40.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:42:40.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:42:40.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:42:40.871 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:42:40.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:42:40.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:42:41.043 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:42:41.042+0000 7f7a7d4c67c0 -1 Falling back to public interface 2024-02-22T02:42:41.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:42:42.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:42:42.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:42:42.264 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:42:42.264 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:42:42.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:42:42.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:42:42.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:42:43.648 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:42:43.650+0000 7f7a7d4c67c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:42:43.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:42:43.679 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:42:43.680 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:42:43.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:42:43.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:42:43.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:42:44.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:42:45.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:42:45.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:42:45.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:42:45.330 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:42:45.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:42:45.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:42:45.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:42:46.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:42:46.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:42:46.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:42:46.755 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:42:46.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:42:46.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:42:47.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:42:48.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:42:48.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:42:48.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:42:48.195 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:42:48.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:42:48.196 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:42:48.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:42:49.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:42:49.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:42:49.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:42:49.616 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:42:49.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:42:49.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:42:50.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:42:51.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:42:51.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:42:51.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:42:51.083 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:42:51.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:42:51.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:42:51.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:42:52.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:42:52.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:42:52.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:42:52.573 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:42:52.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:42:52.573 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:42:53.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:42:54.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:42:54.022 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:42:54.022 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:42:54.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:42:54.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:42:54.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:42:54.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:42:55.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:42:55.492 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:42:55.493 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:42:55.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:42:55.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:42:55.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:42:55.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:42:56.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:42:56.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:42:56.967 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:42:56.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:42:56.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:42:56.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:42:57.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:42:58.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:42:58.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:42:58.437 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:42:58.437 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:42:58.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:42:58.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:42:59.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:00.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:00.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:00.178 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:43:00.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:43:00.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:00.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:01.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:02.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:02.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:02.284 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:43:02.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:43:02.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:02.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:03.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:04.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:04.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:04.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:43:04.093 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:43:04.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:04.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:05.064 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:06.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:06.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:06.066 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:43:06.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:43:06.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:06.067 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:07.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:08.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:08.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:08.035 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:43:08.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:43:08.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:08.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:08.453 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2207081800,v1:127.0.0.1:6803/2207081800] [v2:127.0.0.1:6804/2207081800,v1:127.0.0.1:6805/2207081800] exists,up d33fd331-a230-4b5d-b65f-84908a508883 2024-02-22T02:43:08.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:43:08.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:43:08.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:43:08.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2024-02-22T02:43:08.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-02-22T02:43:08.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-02-22T02:43:08.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=a 2024-02-22T02:43:08.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-02-22T02:43:08.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/a 2024-02-22T02:43:08.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-02-22T02:43:08.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:43:08.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:43:08.456 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:43:08.456 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:43:08.456 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:43:08.456 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:43:08.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-02-22T02:43:08.604 INFO:tasks.workunit.client.0.smithi046.stdout:starting mds.a at 2024-02-22T02:43:08.608 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2024-02-22T02:43:09.120 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'meta' already exists 2024-02-22T02:43:09.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2024-02-22T02:43:10.006 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'data' already exists 2024-02-22T02:43:10.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2024-02-22T02:43:11.226 INFO:tasks.workunit.client.0.smithi046.stderr:filesystem 'myfs' already exists 2024-02-22T02:43:11.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2024-02-22T02:43:16.249 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2024-02-22T02:43:16.632 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-02-22T02:43:16.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2024-02-22T02:43:17.043 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-02-22T02:43:17.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2024-02-22T02:43:17.599 INFO:tasks.workunit.client.0.smithi046.stderr:should be safe to stop [dne] 2024-02-22T02:43:17.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2024-02-22T02:43:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-02-22T02:43:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-02-22T02:43:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=b 2024-02-22T02:43:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-02-22T02:43:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/b 2024-02-22T02:43:17.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-02-22T02:43:17.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:43:17.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:43:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:43:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:43:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:43:17.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:43:17.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-02-22T02:43:17.667 INFO:tasks.workunit.client.0.smithi046.stdout:starting mds.b at 2024-02-22T02:43:17.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2024-02-22T02:43:22.672 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2024-02-22T02:43:23.202 INFO:tasks.workunit.client.0.smithi046.stderr:should be safe to stop [a] 2024-02-22T02:43:23.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2024-02-22T02:43:23.794 INFO:tasks.workunit.client.0.smithi046.stderr:should be safe to stop [b] 2024-02-22T02:43:23.809 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2024-02-22T02:43:24.192 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-02-22T02:43:24.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2024-02-22T02:43:24.767 INFO:tasks.workunit.client.0.smithi046.stderr:should be safe to stop [a,dne1,dne2] 2024-02-22T02:43:24.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2024-02-22T02:43:25.326 INFO:tasks.workunit.client.0.smithi046.stderr:should be safe to stop [b,dne] 2024-02-22T02:43:25.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2024-02-22T02:43:25.709 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-02-22T02:43:25.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2024-02-22T02:43:26.237 INFO:tasks.workunit.client.0.smithi046.stderr:should be safe to stop [dne1,dne2] 2024-02-22T02:43:26.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2024-02-22T02:43:26.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:43:26.253 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:43:26.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:43:26.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:43:26.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:43:26.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:43:26.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-02-22T02:43:26.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:43:26.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:43:26.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:43:26.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:43:26.364 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:43:26.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:43:26.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-02-22T02:43:26.474 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:43:26.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:43:26.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:43:26.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:43:26.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:43:26.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:43:26.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:43:26.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:43:26.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:43:26.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:43:26.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:43:26.482 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:43:26.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:43:26.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:43:26.483 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:43:26.483 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:43:26.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:43:26.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:43:26.484 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:43:26.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:43:26.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:43:26.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:43:26.492 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:43:26.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:43:26.493 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:43:26.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:43:26.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:43:26.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:43:26.494 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-02-22T02:43:26.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-02-22T02:43:26.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-02-22T02:43:26.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-02-22T02:43:26.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:43:26.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:43:26.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:43:26.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:43:26.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:43:26.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:43:26.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:43:26.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:43:26.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:43:26.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:43:26.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:43:26.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:43:26.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:43:26.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:43:26.499 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:43:26.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:43:26.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:43:26.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:43:26.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:43:26.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:43:26.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:43:26.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:43:26.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:43:26.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:43:26.502 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:43:26.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:43:26.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:43:26.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:43:26.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-02-22T02:43:26.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:43:26.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:43:26.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:43:26.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.46785 2024-02-22T02:43:26.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:43:26.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:43:26.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:43:26.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-02-22T02:43:26.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2024-02-22T02:43:26.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2024-02-22T02:43:26.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-02-22T02:43:26.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-02-22T02:43:26.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:43:26.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:43:26.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:43:26.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:43:26.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-02-22T02:43:26.508 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-02-22T02:43:26.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:43:26.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:43:26.901 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:43:26.902 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:43:26.902 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:43:26.902 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:43:26.902 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:43:26.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-02-22T02:43:27.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:43:27.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:43:27.181 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:43:27.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:43:27.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:43:27.182 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:43:27.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:43:27.183 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:43:27.183 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:43:27.183 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:43:27.183 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:43:27.183 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:43:27.183 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:43:27.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:43:27.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get fsid 2024-02-22T02:43:27.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:43:27.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:43:27.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:43:27.298 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:43:27.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:43:27.299 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:43:27.299 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:43:27.299 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:43:27.299 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:43:27.299 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:43:27.299 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:43:27.299 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:43:27.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:43:27.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get mon_host 2024-02-22T02:43:27.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2024-02-22T02:43:27.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-02-22T02:43:27.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:43:27.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:43:27.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:43:27.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-02-22T02:43:27.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:43:28.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:43:28.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:43:28.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:43:28.329 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:43:28.329 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:43:28.329 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:43:28.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:43:28.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:43:28.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:43:28.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2024-02-22T02:43:28.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-02-22T02:43:28.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:43:28.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:43:28.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:43:28.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-02-22T02:43:28.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-02-22T02:43:28.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:43:28.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:43:28.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:43:28.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-02-22T02:43:28.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-02-22T02:43:28.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:43:28.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:43:28.406 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-02-22T02:43:28.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:43:28.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:43:28.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:43:28.406 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:43:28.407 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:43:28.407 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:43:28.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:43:28.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:43:28.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:43:28.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:43:28.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:43:28.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-02-22T02:43:28.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-02-22T02:43:28.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:43:28.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:43:28.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:43:28.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:43:28.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:43:28.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:43:28.408 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-02-22T02:43:28.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:43:28.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a40f73bb-62b5-4dd1-bdc3-3bb414d7fd45 2024-02-22T02:43:28.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a40f73bb-62b5-4dd1-bdc3-3bb414d7fd45' 2024-02-22T02:43:28.409 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 a40f73bb-62b5-4dd1-bdc3-3bb414d7fd45 2024-02-22T02:43:28.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:43:28.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQtNZlro6KGRAAnYFe1YtIUmC9d7Wa4z4vXw== 2024-02-22T02:43:28.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQtNZlro6KGRAAnYFe1YtIUmC9d7Wa4z4vXw=="}' 2024-02-22T02:43:28.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a40f73bb-62b5-4dd1-bdc3-3bb414d7fd45 -i td/ok-to-stop/0/new.json 2024-02-22T02:43:28.741 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:43:28.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-02-22T02:43:28.758 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQtNZlro6KGRAAnYFe1YtIUmC9d7Wa4z4vXw== --osd-uuid a40f73bb-62b5-4dd1-bdc3-3bb414d7fd45 2024-02-22T02:43:28.795 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:43:28.794+0000 7fe1517797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:43:28.834 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:43:28.834+0000 7fe1517797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:43:28.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:43:28.838+0000 7fe1517797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:43:28.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:43:28.838+0000 7fe1517797c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-02-22T02:43:28.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:43:28.838+0000 7fe1517797c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-02-22T02:43:28.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:43:28.838+0000 7fe1517797c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-02-22T02:43:28.874 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:43:28.874+0000 7fe1517797c0 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-02-22T02:43:35.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-02-22T02:43:35.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:43:35.358 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:43:35.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:43:35.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:43:35.922 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:43:35.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:43:35.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:43:35.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:43:35.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:43:35.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:43:35.953 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:43:35.954+0000 7f73b665f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:43:35.982 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:43:35.982+0000 7f73b665f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:43:35.984 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:43:35.986+0000 7f73b665f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:43:36.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:43:36.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:43:36.371 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:43:36.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:43:36.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:43:36.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:36.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:43:36.372 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:43:36.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:36.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:36.581 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:43:36.582+0000 7f73b665f7c0 -1 Falling back to public interface 2024-02-22T02:43:36.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:37.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:37.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:37.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:43:37.794 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:43:37.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:37.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:38.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:39.173 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:43:39.174+0000 7f73b665f7c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:43:39.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:39.201 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:43:39.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:39.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:43:39.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:39.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:39.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:40.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:40.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:40.879 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:43:40.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:43:40.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:40.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:41.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:42.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:42.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:42.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:43:42.323 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:43:42.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:42.324 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:42.768 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:43.769 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:43.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:43.770 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:43:43.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:43:43.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:43.771 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:44.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:45.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:45.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:45.253 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:43:45.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:43:45.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:45.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:45.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:46.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:46.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:46.717 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:43:46.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:43:46.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:46.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:47.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:48.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:48.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:48.139 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:43:48.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:43:48.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:48.140 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:48.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:49.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:49.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:49.592 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:43:49.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:43:49.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:49.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:50.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:51.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:51.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:51.183 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:43:51.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:43:51.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:51.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:51.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:52.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:52.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:52.642 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:43:52.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:43:52.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:52.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:53.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:54.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:54.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:54.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:43:54.642 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:43:54.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:54.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:55.384 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:56.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:56.385 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:56.386 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:43:56.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:43:56.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:56.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:57.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:43:58.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:43:58.242 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:43:58.242 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:43:58.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:43:58.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:43:58.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:43:59.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:00.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:00.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:00.153 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:44:00.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:44:00.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:00.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:44:01.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:02.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:02.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:02.214 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:44:02.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:44:02.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:02.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:44:03.215 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:04.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:04.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:04.217 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:44:04.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:44:04.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:04.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:44:04.687 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1720323418,v1:127.0.0.1:6803/1720323418] [v2:127.0.0.1:6804/1720323418,v1:127.0.0.1:6805/1720323418] exists,up a40f73bb-62b5-4dd1-bdc3-3bb414d7fd45 2024-02-22T02:44:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:44:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:44:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:44:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2024-02-22T02:44:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-02-22T02:44:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:44:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-22T02:44:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:44:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/1 2024-02-22T02:44:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-02-22T02:44:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:44:04.690 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:44:04.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:44:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:44:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:44:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:44:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-02-22T02:44:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-02-22T02:44:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:44:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:44:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:44:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:44:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:44:04.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:44:04.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/1 2024-02-22T02:44:04.693 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:44:04.693 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e172798b-58a7-474d-9fab-87880c9a4fb9 2024-02-22T02:44:04.693 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 e172798b-58a7-474d-9fab-87880c9a4fb9 2024-02-22T02:44:04.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e172798b-58a7-474d-9fab-87880c9a4fb9' 2024-02-22T02:44:04.694 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:44:04.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0tNZlgiHNKhAAJGhTJPA9ZAjMwFvMukXGwg== 2024-02-22T02:44:04.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0tNZlgiHNKhAAJGhTJPA9ZAjMwFvMukXGwg=="}' 2024-02-22T02:44:04.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e172798b-58a7-474d-9fab-87880c9a4fb9 -i td/ok-to-stop/1/new.json 2024-02-22T02:44:05.253 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:44:05.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/1/new.json 2024-02-22T02:44:05.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0tNZlgiHNKhAAJGhTJPA9ZAjMwFvMukXGwg== --osd-uuid e172798b-58a7-474d-9fab-87880c9a4fb9 2024-02-22T02:44:05.305 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:05.305+0000 7fcb5c2f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:44:05.362 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:05.361+0000 7fcb5c2f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:44:05.364 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:05.365+0000 7fcb5c2f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:44:05.364 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:05.365+0000 7fcb5c2f07c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-02-22T02:44:05.364 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:05.365+0000 7fcb5c2f07c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-02-22T02:44:05.364 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:05.365+0000 7fcb5c2f07c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-02-22T02:44:05.486 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:05.489+0000 7fcb5c2f07c0 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2024-02-22T02:44:12.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/1/keyring 2024-02-22T02:44:12.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:44:12.293 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-02-22T02:44:12.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-22T02:44:12.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:44:12.846 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-02-22T02:44:12.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-22T02:44:12.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:44:12.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:44:12.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:44:12.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:44:12.869 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:12.869+0000 7f33871897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:44:12.900 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:12.901+0000 7f33871897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:44:12.902 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:12.901+0000 7f33871897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:44:13.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-22T02:44:13.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:44:13.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-22T02:44:13.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:44:13.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:44:13.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:13.279 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:44:13.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:44:13.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:13.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:13.476 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:13.477+0000 7f33871897c0 -1 Falling back to public interface 2024-02-22T02:44:13.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:14.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:14.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:14.737 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:44:14.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:44:14.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:14.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:15.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:16.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:16.156 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:16.156 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:44:16.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:44:16.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:16.157 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:16.180 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:16.181+0000 7f33871897c0 -1 osd.1 0 log_to_monitors true 2024-02-22T02:44:16.715 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:17.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:17.716 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:17.717 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:44:17.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:44:17.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:17.717 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:18.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:19.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:19.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:19.239 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:44:19.239 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:44:19.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:19.240 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:19.770 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:20.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:20.772 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:20.772 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:44:20.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:44:20.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:20.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:21.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:22.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:22.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:22.287 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:44:22.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:44:22.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:22.287 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:22.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:23.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:23.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:23.727 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:44:23.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:44:23.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:23.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:24.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:25.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:25.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:25.328 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:44:25.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:44:25.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:25.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:25.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:26.775 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:26.775 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:44:26.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:26.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:44:26.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:26.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:27.421 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:28.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:28.423 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:28.423 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:44:28.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:44:28.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:28.424 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:28.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:29.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:29.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:29.868 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:44:29.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:44:29.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:29.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:30.970 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:31.971 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:31.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:31.972 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:44:31.972 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:44:31.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:31.973 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:33.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:34.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:34.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:34.155 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:44:34.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:44:34.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:34.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:34.963 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:35.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:35.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:35.965 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:44:35.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:44:35.965 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:35.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:36.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:37.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:37.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:37.990 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:44:37.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:44:37.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:37.991 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:39.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:40.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:40.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:40.162 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:44:40.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:44:40.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:40.162 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:40.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:41.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:41.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:41.706 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:44:41.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:44:41.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:41.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:44:42.138 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/317339877,v1:127.0.0.1:6811/317339877] [v2:127.0.0.1:6812/317339877,v1:127.0.0.1:6813/317339877] exists,up e172798b-58a7-474d-9fab-87880c9a4fb9 2024-02-22T02:44:42.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:44:42.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:44:42.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:44:42.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2024-02-22T02:44:42.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-02-22T02:44:42.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:44:42.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-22T02:44:42.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:44:42.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/2 2024-02-22T02:44:42.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-02-22T02:44:42.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:44:42.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:44:42.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:44:42.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2024-02-22T02:44:42.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2024-02-22T02:44:42.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:44:42.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:44:42.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-02-22T02:44:42.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:44:42.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:44:42.142 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:44:42.142 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:44:42.142 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:44:42.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/2 2024-02-22T02:44:42.144 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:44:42.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d76dc7a-405f-404c-b01e-48a67fe478cd 2024-02-22T02:44:42.144 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2d76dc7a-405f-404c-b01e-48a67fe478cd' 2024-02-22T02:44:42.145 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 2d76dc7a-405f-404c-b01e-48a67fe478cd 2024-02-22T02:44:42.145 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:44:42.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAatdZlv08jChAATl70LkVfVqmC02gDjBoQ1A== 2024-02-22T02:44:42.165 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAatdZlv08jChAATl70LkVfVqmC02gDjBoQ1A=="}' 2024-02-22T02:44:42.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d76dc7a-405f-404c-b01e-48a67fe478cd -i td/ok-to-stop/2/new.json 2024-02-22T02:44:42.655 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:44:42.670 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/2/new.json 2024-02-22T02:44:42.671 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAatdZlv08jChAATl70LkVfVqmC02gDjBoQ1A== --osd-uuid 2d76dc7a-405f-404c-b01e-48a67fe478cd 2024-02-22T02:44:42.704 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:42.709+0000 7f2f0143b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:44:42.738 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:42.741+0000 7f2f0143b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:44:42.739 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:42.745+0000 7f2f0143b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:44:42.740 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:42.745+0000 7f2f0143b7c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-02-22T02:44:42.740 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:42.745+0000 7f2f0143b7c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-02-22T02:44:42.740 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:42.745+0000 7f2f0143b7c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-02-22T02:44:42.777 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:42.781+0000 7f2f0143b7c0 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2024-02-22T02:44:49.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/2/keyring 2024-02-22T02:44:49.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:44:49.406 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-02-22T02:44:49.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-22T02:44:49.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:44:50.002 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-02-22T02:44:50.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-22T02:44:50.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:44:50.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:44:50.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:44:50.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:44:50.033 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:50.037+0000 7f325d5b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:44:50.035 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:50.041+0000 7f325d5b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:44:50.036 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:50.041+0000 7f325d5b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:44:50.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-22T02:44:50.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:44:50.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-22T02:44:50.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:44:50.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:44:50.458 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:50.459 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:44:50.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:44:50.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:50.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:44:50.598 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:50.601+0000 7f325d5b57c0 -1 Falling back to public interface 2024-02-22T02:44:50.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:51.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:51.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:51.949 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:44:51.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:44:51.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:51.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:44:52.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:53.219 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:44:53.225+0000 7f325d5b57c0 -1 osd.2 0 log_to_monitors true 2024-02-22T02:44:53.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:53.362 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:53.362 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:44:53.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:44:53.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:53.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:44:54.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:55.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:55.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:55.018 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:44:55.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:44:55.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:55.018 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:44:55.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:56.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:56.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:56.500 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:44:56.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:44:56.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:56.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:44:56.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:57.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:57.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:57.956 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:44:57.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:44:57.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:57.957 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:44:58.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:44:59.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:44:59.480 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:44:59.480 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:44:59.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:44:59.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:44:59.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:45:00.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:01.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:01.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:01.053 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:45:01.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:45:01.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:01.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:45:01.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:02.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:02.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:02.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:45:02.488 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:45:02.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:02.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:45:02.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:03.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:03.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:03.949 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:45:03.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:45:03.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:03.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:45:04.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:05.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:05.456 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:05.457 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:45:05.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:45:05.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:05.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:45:05.891 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:06.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:06.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:06.893 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:45:06.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:45:06.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:06.894 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:45:07.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:08.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:08.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:08.601 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:45:08.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:45:08.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:08.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:45:09.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:10.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:10.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:10.379 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:45:10.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:45:10.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:10.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:45:11.266 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:12.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:12.268 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:12.268 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:45:12.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:45:12.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:12.269 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:45:13.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:14.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:14.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:14.107 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:45:14.107 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:45:14.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:14.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:45:15.185 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:16.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:16.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:16.188 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:45:16.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:45:16.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:16.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:45:17.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:18.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:18.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:18.154 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:45:18.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:45:18.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:18.155 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:45:18.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:19.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:19.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:19.698 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-02-22T02:45:19.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-22T02:45:19.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:19.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:45:20.132 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/942523565,v1:127.0.0.1:6819/942523565] [v2:127.0.0.1:6820/942523565,v1:127.0.0.1:6821/942523565] exists,up 2d76dc7a-405f-404c-b01e-48a67fe478cd 2024-02-22T02:45:20.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:45:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:45:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:45:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2024-02-22T02:45:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-02-22T02:45:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:45:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-22T02:45:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:45:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/3 2024-02-22T02:45:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-02-22T02:45:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:45:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:45:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:45:20.136 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:45:20.137 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:45:20.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-02-22T02:45:20.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-02-22T02:45:20.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:45:20.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:45:20.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:45:20.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:45:20.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:45:20.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:45:20.138 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/3 2024-02-22T02:45:20.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:45:20.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d8fcb08c-fccf-46c9-b44d-451eb920273c 2024-02-22T02:45:20.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 d8fcb08c-fccf-46c9-b44d-451eb920273c' 2024-02-22T02:45:20.139 INFO:tasks.workunit.client.0.smithi046.stdout:add osd3 d8fcb08c-fccf-46c9-b44d-451eb920273c 2024-02-22T02:45:20.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:45:20.159 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBAtdZlThLQCRAAi+z3dbcGyNuXeRIB/FKGXg== 2024-02-22T02:45:20.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBAtdZlThLQCRAAi+z3dbcGyNuXeRIB/FKGXg=="}' 2024-02-22T02:45:20.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d8fcb08c-fccf-46c9-b44d-451eb920273c -i td/ok-to-stop/3/new.json 2024-02-22T02:45:20.672 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:45:20.688 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/3/new.json 2024-02-22T02:45:20.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAtdZlThLQCRAAi+z3dbcGyNuXeRIB/FKGXg== --osd-uuid d8fcb08c-fccf-46c9-b44d-451eb920273c 2024-02-22T02:45:20.724 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:45:20.728+0000 7fd4d48f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:45:20.768 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:45:20.772+0000 7fd4d48f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:45:20.769 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:45:20.772+0000 7fd4d48f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:45:20.769 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:45:20.772+0000 7fd4d48f37c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-02-22T02:45:20.769 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:45:20.772+0000 7fd4d48f37c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-02-22T02:45:20.770 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:45:20.772+0000 7fd4d48f37c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-02-22T02:45:20.814 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:45:20.816+0000 7fd4d48f37c0 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2024-02-22T02:45:27.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/3/keyring 2024-02-22T02:45:27.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:45:27.463 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd3 key to auth repository 2024-02-22T02:45:27.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-22T02:45:27.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:45:28.028 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.3 2024-02-22T02:45:28.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-22T02:45:28.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:45:28.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:45:28.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:45:28.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:45:28.052 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:45:28.056+0000 7f1a19aa87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:45:28.054 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:45:28.056+0000 7f1a19aa87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:45:28.055 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:45:28.060+0000 7f1a19aa87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:45:28.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-22T02:45:28.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:45:28.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-22T02:45:28.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:45:28.447 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:45:28.447 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:45:28.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:28.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:45:28.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:28.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:28.620 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:45:28.624+0000 7f1a19aa87c0 -1 Falling back to public interface 2024-02-22T02:45:28.908 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:29.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:29.910 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:29.910 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:45:29.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:45:29.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:29.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:30.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:31.303 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:45:31.308+0000 7f1a19aa87c0 -1 osd.3 0 log_to_monitors true 2024-02-22T02:45:31.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:31.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:31.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:45:31.380 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:45:31.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:31.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:31.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:32.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:32.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:32.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:45:32.844 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:45:32.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:32.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:33.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:34.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:34.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:34.500 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:45:34.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:45:34.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:34.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:34.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:35.952 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:35.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:35.953 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:45:35.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:45:35.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:35.954 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:36.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:37.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:37.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:37.486 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:45:37.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:45:37.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:37.487 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:37.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:38.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:38.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:38.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:45:38.992 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:45:38.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:38.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:39.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:40.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:40.623 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:40.624 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:45:40.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:45:40.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:40.624 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:41.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:42.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:42.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:42.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:45:42.068 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:45:42.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:42.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:42.626 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:43.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:43.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:43.628 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:45:43.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:45:43.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:43.629 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:44.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:45.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:45.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:45.127 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:45:45.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:45:45.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:45.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:45.642 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:46.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:46.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:46.644 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:45:46.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:45:46.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:46.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:47.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:48.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:48.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:48.665 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:45:48.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:45:48.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:48.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:49.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:50.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:50.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:50.602 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:45:50.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:45:50.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:50.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:51.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:52.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:52.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:52.400 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:45:52.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:45:52.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:52.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:53.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:54.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:54.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:54.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:45:54.189 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:45:54.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:54.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:54.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:55.928 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:55.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:55.929 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:45:55.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:45:55.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:55.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:56.898 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:45:57.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:45:57.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:45:57.900 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-02-22T02:45:57.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-22T02:45:57.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:45:57.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-22T02:45:58.381 INFO:tasks.workunit.client.0.smithi046.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2703012855,v1:127.0.0.1:6827/2703012855] [v2:127.0.0.1:6828/2703012855,v1:127.0.0.1:6829/2703012855] exists,up d8fcb08c-fccf-46c9-b44d-451eb920273c 2024-02-22T02:45:58.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:45:58.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:45:58.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:45:58.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-02-22T02:45:59.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2024-02-22T02:46:00.012 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'ec' already exists 2024-02-22T02:46:00.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2024-02-22T02:46:00.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-22T02:46:00.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-22T02:46:00.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-22T02:46:00.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-22T02:46:00.030 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-22T02:46:00.030 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-22T02:46:00.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-22T02:46:00.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-22T02:46:00.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-22T02:46:00.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-22T02:46:00.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-22T02:46:00.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-22T02:46:00.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-22T02:46:00.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-22T02:46:00.119 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-22T02:46:00.985 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-22T02:46:00.985 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-02-22T02:46:00.986 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-02-22T02:46:00.986 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-02-22T02:46:00.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-22T02:46:00.986 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:00.986 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-22T02:46:01.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-02-22T02:46:01.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-02-22T02:46:01.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-02-22T02:46:01.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:01.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-22T02:46:01.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-02-22T02:46:01.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-02-22T02:46:01.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672979' 2024-02-22T02:46:01.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:01.629 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-22T02:46:01.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-02-22T02:46:01.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-02-22T02:46:01.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672979 2-64424509454' 2024-02-22T02:46:01.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:01.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-22T02:46:02.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345928 2024-02-22T02:46:02.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345928 2024-02-22T02:46:02.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672979 2-64424509454 3-85899345928' 2024-02-22T02:46:02.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:02.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-02-22T02:46:02.063 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:02.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-22T02:46:02.064 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-02-22T02:46:02.064 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:02.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-02-22T02:46:02.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-02-22T02:46:02.066 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836513 2024-02-22T02:46:02.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-22T02:46:02.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836513 2024-02-22T02:46:02.482 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-22T02:46:03.484 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-22T02:46:03.484 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-22T02:46:03.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836513 2024-02-22T02:46:03.907 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:03.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-02-22T02:46:03.908 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:03.909 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-22T02:46:03.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-02-22T02:46:03.910 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:03.911 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672979 2024-02-22T02:46:03.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-02-22T02:46:03.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-02-22T02:46:03.912 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-22T02:46:04.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672979 2024-02-22T02:46:04.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:04.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-02-22T02:46:04.331 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:04.333 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-22T02:46:04.334 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-02-22T02:46:04.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:04.336 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509454 2024-02-22T02:46:04.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-02-22T02:46:04.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-02-22T02:46:04.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-22T02:46:04.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-02-22T02:46:04.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:04.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345928 2024-02-22T02:46:04.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:04.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-22T02:46:04.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345928 2024-02-22T02:46:04.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:04.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345928 2024-02-22T02:46:04.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2024-02-22T02:46:04.753 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345928 2024-02-22T02:46:04.754 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-22T02:46:05.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345929 -lt 85899345928 2024-02-22T02:46:05.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-22T02:46:05.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-22T02:46:05.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-22T02:46:05.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-02-22T02:46:05.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-22T02:46:05.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-22T02:46:05.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-22T02:46:05.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-22T02:46:05.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-22T02:46:05.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-22T02:46:05.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-22T02:46:06.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-02-22T02:46:06.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-22T02:46:06.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-22T02:46:06.115 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-22T02:46:06.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-02-22T02:46:06.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-22T02:46:06.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-22T02:46:06.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2024-02-22T02:46:07.303 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 min_size to 3 2024-02-22T02:46:07.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2024-02-22T02:46:07.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-22T02:46:07.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-22T02:46:07.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-22T02:46:07.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-22T02:46:07.326 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-22T02:46:07.326 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-22T02:46:07.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-22T02:46:07.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-22T02:46:07.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-22T02:46:07.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-22T02:46:07.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-22T02:46:07.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-22T02:46:07.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-22T02:46:07.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-22T02:46:07.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-22T02:46:07.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-22T02:46:07.874 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-02-22T02:46:07.874 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-02-22T02:46:07.874 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-02-22T02:46:07.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-22T02:46:07.874 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:07.874 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-22T02:46:08.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-02-22T02:46:08.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-02-22T02:46:08.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-02-22T02:46:08.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:08.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-22T02:46:08.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-02-22T02:46:08.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-02-22T02:46:08.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672983' 2024-02-22T02:46:08.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:08.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-22T02:46:08.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-02-22T02:46:08.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-02-22T02:46:08.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672983 2-64424509457' 2024-02-22T02:46:08.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:08.355 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-22T02:46:08.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345932 2024-02-22T02:46:08.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345932 2024-02-22T02:46:08.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672983 2-64424509457 3-85899345932' 2024-02-22T02:46:08.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:08.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-02-22T02:46:08.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:08.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-22T02:46:08.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-02-22T02:46:08.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:08.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-02-22T02:46:08.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-02-22T02:46:08.509 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836516 2024-02-22T02:46:08.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-22T02:46:08.933 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836516 2024-02-22T02:46:08.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-22T02:46:09.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-22T02:46:09.936 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-22T02:46:10.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836516 2024-02-22T02:46:10.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:10.376 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-02-22T02:46:10.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:10.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-22T02:46:10.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-02-22T02:46:10.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:10.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-02-22T02:46:10.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-02-22T02:46:10.381 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672983 2024-02-22T02:46:10.381 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-22T02:46:10.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-02-22T02:46:10.803 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:10.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-02-22T02:46:10.803 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:10.804 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-22T02:46:10.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-02-22T02:46:10.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:10.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-02-22T02:46:10.807 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509457 2024-02-22T02:46:10.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-02-22T02:46:10.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-22T02:46:11.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-02-22T02:46:11.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:11.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345932 2024-02-22T02:46:11.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:11.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-22T02:46:11.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345932 2024-02-22T02:46:11.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:11.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345932 2024-02-22T02:46:11.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-02-22T02:46:11.230 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345932 2024-02-22T02:46:11.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-22T02:46:11.683 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345932 -lt 85899345932 2024-02-22T02:46:11.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-22T02:46:11.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-22T02:46:11.684 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-22T02:46:12.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-02-22T02:46:12.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-22T02:46:12.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-22T02:46:12.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-22T02:46:12.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-22T02:46:12.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-22T02:46:12.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-22T02:46:12.248 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-22T02:46:12.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-02-22T02:46:12.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-22T02:46:12.678 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-22T02:46:12.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-22T02:46:13.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-02-22T02:46:13.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-22T02:46:13.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-22T02:46:13.252 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2024-02-22T02:46:13.663 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:13.680 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2024-02-22T02:46:14.076 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:14.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2024-02-22T02:46:14.491 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:14.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2024-02-22T02:46:14.896 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:14.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-02-22T02:46:15.211 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:15.212 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-02-22T02:46:15.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-02-22T02:46:15.519 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:15.519 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-02-22T02:46:15.523 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-02-22T02:46:15.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2024-02-22T02:46:15.934 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:15.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2024-02-22T02:46:15.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2024-02-22T02:46:16.335 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:16.335 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2024-02-22T02:46:17.229 INFO:tasks.workunit.client.0.smithi046.stderr:set pool 1 min_size to 2 2024-02-22T02:46:17.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2024-02-22T02:46:17.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-22T02:46:17.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-22T02:46:17.253 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-22T02:46:17.254 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-22T02:46:17.254 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-22T02:46:17.254 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-22T02:46:17.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-22T02:46:17.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-22T02:46:17.255 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-22T02:46:17.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-22T02:46:17.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-22T02:46:17.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-22T02:46:17.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-22T02:46:17.378 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-22T02:46:17.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-22T02:46:18.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-22T02:46:18.160 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-02-22T02:46:18.160 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-02-22T02:46:18.160 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-02-22T02:46:18.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-22T02:46:18.160 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:18.161 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-22T02:46:18.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836520 2024-02-22T02:46:18.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836520 2024-02-22T02:46:18.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520' 2024-02-22T02:46:18.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:18.323 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-22T02:46:18.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-02-22T02:46:18.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-02-22T02:46:18.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672987' 2024-02-22T02:46:18.541 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:18.542 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-22T02:46:18.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-02-22T02:46:18.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-02-22T02:46:18.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672987 2-64424509461' 2024-02-22T02:46:18.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:18.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-22T02:46:18.846 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345936 2024-02-22T02:46:18.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345936 2024-02-22T02:46:18.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672987 2-64424509461 3-85899345936' 2024-02-22T02:46:18.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:18.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836520 2024-02-22T02:46:18.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:18.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-22T02:46:18.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836520 2024-02-22T02:46:18.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:18.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836520 2024-02-22T02:46:18.850 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.0 seq 21474836520 2024-02-22T02:46:18.850 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-02-22T02:46:18.851 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-22T02:46:19.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836520 2024-02-22T02:46:19.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-22T02:46:20.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-22T02:46:20.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-22T02:46:20.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836520 2024-02-22T02:46:20.742 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:20.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-02-22T02:46:20.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:20.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-22T02:46:20.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-02-22T02:46:20.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:20.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-02-22T02:46:20.747 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672987 2024-02-22T02:46:20.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-02-22T02:46:20.748 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-22T02:46:21.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672987 2024-02-22T02:46:21.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:21.170 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-02-22T02:46:21.171 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:21.171 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-22T02:46:21.172 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-02-22T02:46:21.173 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:21.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-02-22T02:46:21.174 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509461 2024-02-22T02:46:21.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-02-22T02:46:21.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-22T02:46:21.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509461 2024-02-22T02:46:21.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:21.602 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345936 2024-02-22T02:46:21.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:21.604 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-22T02:46:21.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345936 2024-02-22T02:46:21.605 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:21.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345936 2024-02-22T02:46:21.606 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345936' 2024-02-22T02:46:21.606 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345936 2024-02-22T02:46:21.606 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-22T02:46:22.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345936 -lt 85899345936 2024-02-22T02:46:22.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-22T02:46:22.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-22T02:46:22.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-22T02:46:22.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-02-22T02:46:22.630 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-22T02:46:22.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-22T02:46:22.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-22T02:46:22.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-22T02:46:22.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-22T02:46:22.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-22T02:46:22.630 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-22T02:46:23.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-02-22T02:46:23.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-22T02:46:23.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-22T02:46:23.026 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-22T02:46:23.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-02-22T02:46:23.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-22T02:46:23.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-22T02:46:23.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-02-22T02:46:23.952 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:23.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-02-22T02:46:24.361 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:24.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-02-22T02:46:24.702 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:24.703 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-02-22T02:46:24.706 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-02-22T02:46:25.004 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:25.004 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-02-22T02:46:25.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-02-22T02:46:25.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2024-02-22T02:46:25.432 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:25.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2024-02-22T02:46:25.432 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2024-02-22T02:46:25.888 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:25.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2024-02-22T02:46:25.889 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2024-02-22T02:46:26.308 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:26.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2024-02-22T02:46:26.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2024-02-22T02:46:26.721 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:26.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2024-02-22T02:46:26.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:46:26.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:46:26.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:46:26.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:46:26.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:46:27.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:46:27.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2024-02-22T02:46:28.016 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 is already down. 2024-02-22T02:46:28.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2024-02-22T02:46:28.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-02-22T02:46:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-02-22T02:46:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-02-22T02:46:28.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-02-22T02:46:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-22T02:46:28.035 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-22T02:46:28.036 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-22T02:46:28.036 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-22T02:46:28.036 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-22T02:46:28.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-22T02:46:28.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-02-22T02:46:28.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-02-22T02:46:28.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-02-22T02:46:28.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-22T02:46:28.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-22T02:46:28.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-22T02:46:28.667 INFO:tasks.workunit.client.0.smithi046.stderr:1 2024-02-22T02:46:28.667 INFO:tasks.workunit.client.0.smithi046.stderr:2 2024-02-22T02:46:28.667 INFO:tasks.workunit.client.0.smithi046.stderr:3' 2024-02-22T02:46:28.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-22T02:46:28.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:28.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-22T02:46:28.788 INFO:tasks.workunit.client.0.smithi046.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-22T02:46:28.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-02-22T02:46:28.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-02-22T02:46:28.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-02-22T02:46:28.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:28.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-22T02:46:28.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672991 2024-02-22T02:46:28.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672991 2024-02-22T02:46:28.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 1-42949672991' 2024-02-22T02:46:28.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:28.983 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-22T02:46:29.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-02-22T02:46:29.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-02-22T02:46:29.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 1-42949672991 2-64424509465' 2024-02-22T02:46:29.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-22T02:46:29.135 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-22T02:46:29.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345940 2024-02-22T02:46:29.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345940 2024-02-22T02:46:29.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 1-42949672991 2-64424509465 3-85899345940' 2024-02-22T02:46:29.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:29.295 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672991 2024-02-22T02:46:29.295 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:29.296 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-22T02:46:29.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672991 2024-02-22T02:46:29.297 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:29.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672991 2024-02-22T02:46:29.298 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-02-22T02:46:29.299 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.1 seq 42949672991 2024-02-22T02:46:29.299 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-22T02:46:29.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672991 -lt 42949672991 2024-02-22T02:46:29.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:29.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-02-22T02:46:29.723 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:29.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-22T02:46:29.726 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-02-22T02:46:29.726 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:29.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-02-22T02:46:29.727 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.2 seq 64424509465 2024-02-22T02:46:29.728 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-02-22T02:46:29.728 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-22T02:46:30.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-02-22T02:46:30.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-22T02:46:30.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345940 2024-02-22T02:46:30.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-22T02:46:30.152 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-22T02:46:30.152 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345940 2024-02-22T02:46:30.153 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-22T02:46:30.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345940 2024-02-22T02:46:30.154 INFO:tasks.workunit.client.0.smithi046.stdout:waiting osd.3 seq 85899345940 2024-02-22T02:46:30.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345940' 2024-02-22T02:46:30.155 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-22T02:46:30.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345938 -lt 85899345940 2024-02-22T02:46:30.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-22T02:46:31.558 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-22T02:46:31.558 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-22T02:46:31.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345940 -lt 85899345940 2024-02-22T02:46:31.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-02-22T02:46:31.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-22T02:46:31.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-22T02:46:32.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 32 == 0 2024-02-22T02:46:32.539 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-02-22T02:46:32.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-02-22T02:46:32.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-02-22T02:46:32.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-02-22T02:46:32.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-02-22T02:46:32.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-02-22T02:46:32.539 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-02-22T02:46:33.001 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=32 2024-02-22T02:46:33.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-02-22T02:46:33.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-22T02:46:33.002 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-22T02:46:33.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 32 = 32 2024-02-22T02:46:33.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-02-22T02:46:33.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-02-22T02:46:33.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2024-02-22T02:46:33.947 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2024-02-22T02:46:33.962 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-02-22T02:46:34.345 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:34.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-02-22T02:46:34.649 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:34.649 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-02-22T02:46:34.653 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-02-22T02:46:34.951 INFO:tasks.workunit.client.0.smithi046.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-02-22T02:46:34.952 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-02-22T02:46:34.956 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-02-22T02:46:34.957 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:46:34.958 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:46:34.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:46:34.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:46:34.958 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:46:35.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:46:35.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-02-22T02:46:35.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:46:35.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:46:35.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:46:35.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:46:35.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:46:35.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:46:35.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:46:35.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:46:35.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:46:35.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:46:35.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:46:35.282 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:46:35.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:46:35.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:46:35.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:46:35.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:46:35.284 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:46:35.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:46:35.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:46:35.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:46:35.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:46:35.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:46:35.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:46:35.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:35.314 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:35.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:46:35.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:46:35.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:46:35.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-02-22T02:46:35.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-02-22T02:46:35.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-02-22T02:46:35.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-02-22T02:46:35.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:46:35.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:46:35.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:46:35.316 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:46:35.317 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:46:35.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:46:35.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:46:35.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:46:35.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:46:35.318 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:46:35.318 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:46:35.319 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:46:35.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:46:35.320 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:46:35.320 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:46:35.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:46:35.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:46:35.321 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:46:35.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:46:35.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:46:35.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:46:35.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:46:35.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:46:35.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:35.325 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:35.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:46:35.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:46:35.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:46:35.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-02-22T02:46:35.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:46:35.326 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:35.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:35.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.46785 2024-02-22T02:46:35.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:46:35.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:46:35.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:46:35.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-02-22T02:46:35.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2024-02-22T02:46:35.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2024-02-22T02:46:35.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-02-22T02:46:35.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-02-22T02:46:35.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:46:35.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:46:35.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:46:35.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:46:35.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-02-22T02:46:35.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-02-22T02:46:35.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:46:35.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:46:35.720 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:46:35.721 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:46:35.721 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:35.721 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:35.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:46:35.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-02-22T02:46:35.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:46:35.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:46:35.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:46:35.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:46:35.932 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:46:35.933 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:46:35.934 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:46:35.934 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:46:35.934 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:46:35.934 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:46:35.934 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:35.935 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:35.935 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:46:35.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:46:35.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get fsid 2024-02-22T02:46:36.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:46:36.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:46:36.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:46:36.050 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:46:36.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:46:36.051 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:46:36.051 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:46:36.051 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:46:36.052 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:46:36.052 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:36.052 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:36.052 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:46:36.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:46:36.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get mon_host 2024-02-22T02:46:36.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2024-02-22T02:46:36.992 INFO:tasks.workunit.client.0.smithi046.stderr:invalid mon(s) specified: [dne] 2024-02-22T02:46:37.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2024-02-22T02:46:37.188 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2024-02-22T02:46:37.192 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2024-02-22T02:46:37.420 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-02-22T02:46:37.425 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2024-02-22T02:46:37.646 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: removing mon.a would break quorum 2024-02-22T02:46:37.651 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2024-02-22T02:46:37.902 INFO:tasks.workunit.client.0.smithi046.stderr:mon.dne does not exist 2024-02-22T02:46:37.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-02-22T02:46:37.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:46:37.919 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:46:37.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:46:37.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:46:37.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:46:38.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:46:38.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-02-22T02:46:38.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:46:38.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:46:38.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:46:38.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:46:38.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:46:38.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:46:38.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:46:38.028 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:46:38.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:46:38.034 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:46:38.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:46:38.035 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:46:38.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:46:38.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:46:38.037 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:46:38.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:46:38.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:46:38.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:46:38.039 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:46:38.039 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:46:38.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:46:38.041 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:46:38.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:46:38.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:38.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:38.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:46:38.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:46:38.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:46:38.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-02-22T02:46:38.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-02-22T02:46:38.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-02-22T02:46:38.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-02-22T02:46:38.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:46:38.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:46:38.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:46:38.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:46:38.045 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:46:38.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:46:38.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:46:38.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:46:38.047 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:46:38.048 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:46:38.048 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:46:38.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:46:38.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:46:38.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:46:38.051 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:46:38.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:46:38.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:46:38.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:46:38.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:46:38.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:46:38.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:46:38.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:46:38.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:46:38.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:38.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:38.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:46:38.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:46:38.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:46:38.057 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-02-22T02:46:38.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:46:38.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:38.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:38.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.46785 2024-02-22T02:46:38.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:46:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:46:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:46:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-02-22T02:46:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2024-02-22T02:46:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2024-02-22T02:46:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2024-02-22T02:46:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-02-22T02:46:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:46:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:46:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:46:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:46:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-02-22T02:46:38.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-02-22T02:46:38.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:46:38.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:46:38.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:46:38.438 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:46:38.438 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:38.438 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:38.438 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:46:38.438 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-02-22T02:46:38.657 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:46:38.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:46:38.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:46:38.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:46:38.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:46:38.657 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:46:38.657 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:46:38.657 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:46:38.657 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:46:38.657 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:46:38.657 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:38.658 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:38.658 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:46:38.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:46:38.658 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get fsid 2024-02-22T02:46:38.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:46:38.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:46:38.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:46:38.773 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:46:38.774 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:46:38.774 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:46:38.774 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:46:38.774 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:46:38.774 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:46:38.774 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:38.774 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:38.775 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:46:38.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:46:38.775 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get mon_host 2024-02-22T02:46:38.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-02-22T02:46:38.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:46:38.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:46:38.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-02-22T02:46:38.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:46:38.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-02-22T02:46:38.886 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-02-22T02:46:39.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:46:39.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:46:39.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:46:39.274 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:46:39.274 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:39.274 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:39.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:46:39.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-02-22T02:46:39.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:46:39.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-02-22T02:46:39.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:46:39.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-02-22T02:46:39.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:46:39.495 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:46:39.496 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-02-22T02:46:39.496 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-02-22T02:46:39.496 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-02-22T02:46:39.497 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:46:39.497 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:39.497 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:39.497 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.b.asok 2024-02-22T02:46:39.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:46:39.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.b.asok config get fsid 2024-02-22T02:46:39.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-02-22T02:46:39.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:46:39.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-02-22T02:46:39.615 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:46:39.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:46:39.616 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-02-22T02:46:39.616 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-02-22T02:46:39.616 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-02-22T02:46:39.617 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:46:39.617 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:39.617 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:39.617 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.b.asok 2024-02-22T02:46:39.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:46:39.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.b.asok config get mon_host 2024-02-22T02:46:39.730 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2024-02-22T02:46:42.963 INFO:tasks.workunit.client.0.smithi046.stderr:invalid mon(s) specified: [dne] 2024-02-22T02:46:42.979 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2024-02-22T02:46:43.267 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2024-02-22T02:46:43.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2024-02-22T02:46:43.460 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2024-02-22T02:46:43.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2024-02-22T02:46:43.756 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2024-02-22T02:46:43.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2024-02-22T02:46:44.037 INFO:tasks.workunit.client.0.smithi046.stderr:adding another mon that is not yet online will not break quorum 2024-02-22T02:46:44.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2024-02-22T02:46:44.424 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.a 2024-02-22T02:46:44.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2024-02-22T02:46:44.726 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.b 2024-02-22T02:46:44.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2024-02-22T02:46:45.071 INFO:tasks.workunit.client.0.smithi046.stderr:mon.dne does not exist 2024-02-22T02:46:45.088 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-02-22T02:46:45.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:46:45.088 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:46:45.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:46:45.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:46:45.089 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:46:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:46:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-02-22T02:46:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:46:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:46:45.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:46:45.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:46:45.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:46:45.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:46:45.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:46:45.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:46:45.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:46:45.204 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:46:45.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:46:45.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:46:45.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:46:45.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:46:45.207 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:46:45.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:46:45.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:46:45.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:46:45.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:46:45.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:46:45.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:46:45.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:46:45.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:46:45.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:45.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:45.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:46:45.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:46:45.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:46:45.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-02-22T02:46:45.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-02-22T02:46:45.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-02-22T02:46:45.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-02-22T02:46:45.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:46:45.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:46:45.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:46:45.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:46:45.217 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:46:45.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:46:45.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:46:45.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:46:45.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:46:45.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:46:45.219 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:46:45.219 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:46:45.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:46:45.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:46:45.220 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:46:45.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:46:45.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:46:45.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:46:45.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:46:45.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:46:45.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:46:45.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:46:45.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:46:45.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:45.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:45.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:46:45.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:46:45.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:46:45.227 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-02-22T02:46:45.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:46:45.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:45.228 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:45.228 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.46785 2024-02-22T02:46:45.230 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:46:45.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:46:45.231 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:46:45.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-02-22T02:46:45.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2024-02-22T02:46:45.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2024-02-22T02:46:45.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2024-02-22T02:46:45.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-02-22T02:46:45.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:46:45.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:46:45.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:46:45.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:46:45.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-02-22T02:46:45.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-02-22T02:46:45.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:46:45.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:46:45.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:46:45.636 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:46:45.636 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:45.636 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:45.636 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:46:45.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-02-22T02:46:45.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:46:45.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:46:45.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:46:45.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:46:45.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:46:45.841 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:46:45.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:46:45.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:46:45.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:46:45.842 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:46:45.842 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:45.842 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:45.842 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:46:45.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:46:45.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get fsid 2024-02-22T02:46:45.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:46:45.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:46:45.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:46:45.960 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:46:45.961 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:46:45.961 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:46:45.961 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:46:45.961 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:46:45.962 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:46:45.962 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:45.962 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:45.962 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:46:45.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:46:45.962 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get mon_host 2024-02-22T02:46:46.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-02-22T02:46:46.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:46:46.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:46:46.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-02-22T02:46:46.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:46:46.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-02-22T02:46:46.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-02-22T02:46:46.479 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:46:46.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:46:46.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:46:46.480 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:46:46.481 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:46.481 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:46.481 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:46:46.481 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-02-22T02:46:46.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:46:46.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-02-22T02:46:46.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:46:46.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-02-22T02:46:46.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:46:46.710 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:46:46.710 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-02-22T02:46:46.710 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-02-22T02:46:46.710 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-02-22T02:46:46.711 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:46:46.711 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:46.711 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:46.711 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.b.asok 2024-02-22T02:46:46.711 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:46:46.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.b.asok config get fsid 2024-02-22T02:46:46.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-02-22T02:46:46.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:46:46.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-02-22T02:46:46.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:46:46.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:46:46.832 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-02-22T02:46:46.832 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-02-22T02:46:46.832 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-02-22T02:46:46.833 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:46:46.833 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:46.833 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:46.833 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.b.asok 2024-02-22T02:46:46.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:46:46.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.b.asok config get mon_host 2024-02-22T02:46:46.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-02-22T02:46:46.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:46:46.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:46:46.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-02-22T02:46:46.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:46:46.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-02-22T02:46:46.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-02-22T02:46:47.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:46:47.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:46:47.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:46:47.599 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:46:47.599 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:47.599 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:47.599 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:46:47.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-02-22T02:46:47.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:46:47.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-02-22T02:46:47.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:46:47.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-02-22T02:46:47.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:46:47.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:46:47.833 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-02-22T02:46:47.833 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-02-22T02:46:47.833 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-02-22T02:46:47.833 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:46:47.834 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:47.834 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:47.834 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.c.asok 2024-02-22T02:46:47.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:46:47.834 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.c.asok config get fsid 2024-02-22T02:46:47.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-02-22T02:46:47.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:46:47.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-02-22T02:46:47.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:46:47.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:46:47.952 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-02-22T02:46:47.952 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-02-22T02:46:47.952 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-02-22T02:46:47.953 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:46:47.953 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:46:47.953 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:46:47.953 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.c.asok 2024-02-22T02:46:47.953 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:46:47.953 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.c.asok config get mon_host 2024-02-22T02:46:48.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2024-02-22T02:46:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-02-22T02:46:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-02-22T02:46:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-02-22T02:46:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-02-22T02:46:48.076 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-02-22T02:46:48.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-02-22T02:46:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1708570068 2024-02-22T02:46:48.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:46:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570008 -lt 1708570068 ]] 2024-02-22T02:46:48.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:46:48.081 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:46:51.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:46:51.323 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:51.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:51.324 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:46:51.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-02-22T02:46:51.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-02-22T02:46:51.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-02-22T02:46:51.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-02-22T02:46:51.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-02-22T02:46:51.361 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2024-02-22T02:46:51.674 INFO:tasks.workunit.client.0.smithi046.stderr:invalid mon(s) specified: [dne] 2024-02-22T02:46:51.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-02-22T02:46:52.020 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (b,c) after stopping [a] 2024-02-22T02:46:52.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-02-22T02:46:52.351 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (a,c) after stopping [b] 2024-02-22T02:46:52.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-02-22T02:46:52.687 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (a,b) after stopping [c] 2024-02-22T02:46:52.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2024-02-22T02:46:52.992 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2024-02-22T02:46:52.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2024-02-22T02:46:53.290 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2024-02-22T02:46:53.295 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2024-02-22T02:46:53.524 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2024-02-22T02:46:53.529 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-02-22T02:46:53.791 INFO:tasks.workunit.client.0.smithi046.stderr:adding another mon that is not yet online will not break quorum 2024-02-22T02:46:53.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-02-22T02:46:54.254 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.a 2024-02-22T02:46:54.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-02-22T02:46:54.581 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.b 2024-02-22T02:46:54.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-02-22T02:46:54.880 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.c 2024-02-22T02:46:54.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2024-02-22T02:46:54.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:46:54.898 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:46:54.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:46:54.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:46:54.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:46:55.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:46:55.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2024-02-22T02:46:55.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-02-22T02:46:55.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=2 2024-02-22T02:46:55.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-02-22T02:46:55.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 2 ]] 2024-02-22T02:46:55.007 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-02-22T02:46:55.007 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-02-22T02:46:55.008 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1708570075 2024-02-22T02:46:55.008 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:46:55.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570015 -lt 1708570075 ]] 2024-02-22T02:46:55.009 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:46:55.009 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:46:55.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:46:55.252 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:55.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:46:55.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:55.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:46:55.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:46:55.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:46:55.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570015 -lt 1708570075 ]] 2024-02-22T02:46:55.286 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:46:55.287 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:46:55.524 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:46:55.524 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:55.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:46:55.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:55.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:46:55.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:46:55.555 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:46:55.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570015 -lt 1708570075 ]] 2024-02-22T02:46:55.556 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:46:55.557 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:46:55.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:46:55.783 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:55.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:55.784 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:46:55.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:46:55.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:46:55.821 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:46:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570015 -lt 1708570075 ]] 2024-02-22T02:46:55.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:46:55.822 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:46:56.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:46:56.056 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:56.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:56.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:46:56.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:46:56.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:46:56.092 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:46:56.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570016 -lt 1708570075 ]] 2024-02-22T02:46:56.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:46:56.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:46:56.325 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:46:56.326 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:56.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:56.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:46:56.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:46:56.365 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:46:56.365 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:46:56.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570016 -lt 1708570075 ]] 2024-02-22T02:46:56.366 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:46:56.366 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:46:56.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:46:56.596 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:56.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:56.597 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:46:56.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:46:56.633 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:46:56.633 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:46:56.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570016 -lt 1708570075 ]] 2024-02-22T02:46:56.634 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:46:56.635 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:46:56.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:46:56.868 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:56.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:46:56.869 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:56.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:46:56.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:46:56.904 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:46:56.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570016 -lt 1708570075 ]] 2024-02-22T02:46:56.906 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:46:56.906 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:46:57.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:46:57.146 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:57.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:57.147 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:46:57.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:46:57.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:46:57.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:46:57.183 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570017 -lt 1708570075 ]] 2024-02-22T02:46:57.184 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:46:57.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:46:57.422 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:46:57.423 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:57.423 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:57.424 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:46:57.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:46:57.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:46:57.460 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:46:57.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570017 -lt 1708570075 ]] 2024-02-22T02:46:57.461 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:46:57.461 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:46:57.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:46:57.696 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:57.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:46:57.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:46:57.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:46:57.732 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:46:57.732 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:46:57.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570017 -lt 1708570075 ]] 2024-02-22T02:46:57.733 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:46:57.734 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:00.964 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:00.964 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:00.964 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:47:00.965 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:00.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:00.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:00.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:00.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570021 -lt 1708570075 ]] 2024-02-22T02:47:00.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:47:00.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:01.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:01.210 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:01.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:47:01.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:01.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:01.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:01.237 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:01.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570021 -lt 1708570075 ]] 2024-02-22T02:47:01.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:47:01.238 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:01.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:01.454 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:01.454 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:47:01.455 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:01.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:01.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:01.488 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:01.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570021 -lt 1708570075 ]] 2024-02-22T02:47:01.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:47:01.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:01.707 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:01.707 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:01.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:01.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:47:01.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:01.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:01.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:01.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570021 -lt 1708570075 ]] 2024-02-22T02:47:01.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-02-22T02:47:01.744 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:10.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:10.079 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":6,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:10.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-02-22T02:47:10.080 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:46:45.262317Z","created":"2024-02-22T02:46:45.262317Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:10.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-02-22T02:47:10.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-02-22T02:47:10.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-02-22T02:47:10.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-02-22T02:47:10.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-02-22T02:47:10.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-02-22T02:47:13.306 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2024-02-22T02:47:13.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-02-22T02:47:13.602 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (a,c) after stopping [b] 2024-02-22T02:47:13.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-02-22T02:47:13.922 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2024-02-22T02:47:13.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-02-22T02:47:14.164 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-02-22T02:47:14.169 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-02-22T02:47:14.423 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: removing mon.a would break quorum 2024-02-22T02:47:14.428 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-02-22T02:47:17.661 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.b 2024-02-22T02:47:17.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-02-22T02:47:17.918 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: removing mon.c would break quorum 2024-02-22T02:47:17.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-02-22T02:47:17.922 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:47:17.922 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:47:17.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:47:17.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:47:17.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:47:18.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:47:18.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-02-22T02:47:18.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:47:18.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:47:18.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:47:18.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:47:18.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:47:18.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:47:18.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:47:18.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:47:18.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:47:18.040 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:47:18.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:47:18.041 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:47:18.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:47:18.042 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:47:18.042 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:47:18.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:47:18.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:47:18.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:47:18.043 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:47:18.044 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:47:18.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:47:18.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:47:18.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:47:18.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:18.049 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:18.050 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:47:18.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:47:18.051 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:47:18.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-02-22T02:47:18.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-02-22T02:47:18.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-02-22T02:47:18.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-02-22T02:47:18.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:47:18.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:47:18.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:47:18.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:47:18.052 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:47:18.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:47:18.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:47:18.052 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:47:18.054 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:47:18.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:47:18.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:47:18.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:47:18.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:47:18.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:47:18.057 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:47:18.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:47:18.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:47:18.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:47:18.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:47:18.059 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:47:18.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:47:18.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:47:18.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:47:18.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:18.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:18.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:47:18.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:47:18.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:47:18.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-02-22T02:47:18.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:47:18.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:18.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:18.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.46785 2024-02-22T02:47:18.068 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:47:18.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:47:18.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:47:18.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-02-22T02:47:18.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2024-02-22T02:47:18.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2024-02-22T02:47:18.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2024-02-22T02:47:18.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-02-22T02:47:18.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:47:18.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:47:18.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:47:18.068 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:47:18.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-02-22T02:47:18.069 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-02-22T02:47:18.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:47:18.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:47:18.475 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:47:18.475 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:47:18.476 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:18.476 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:18.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:47:18.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-02-22T02:47:18.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:47:18.705 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:47:18.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:47:18.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:47:18.706 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:47:18.707 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:47:18.707 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:47:18.707 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:47:18.707 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:47:18.707 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:47:18.707 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:18.707 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:18.707 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:47:18.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:47:18.708 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get fsid 2024-02-22T02:47:18.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:47:18.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:47:18.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:47:18.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:47:18.825 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:47:18.825 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:47:18.826 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:47:18.826 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:47:18.826 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:47:18.826 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:18.827 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:18.827 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:47:18.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:47:18.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get mon_host 2024-02-22T02:47:18.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-02-22T02:47:18.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:47:18.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:47:18.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-02-22T02:47:18.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:47:18.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-02-22T02:47:18.940 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-02-22T02:47:19.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:47:19.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:47:19.362 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:47:19.362 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:47:19.363 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:19.363 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:19.363 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:47:19.363 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-02-22T02:47:19.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:47:19.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-02-22T02:47:19.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:47:19.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-02-22T02:47:19.591 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:47:19.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:47:19.592 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-02-22T02:47:19.592 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-02-22T02:47:19.592 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-02-22T02:47:19.593 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:47:19.593 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:19.593 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:19.593 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.b.asok 2024-02-22T02:47:19.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:47:19.594 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.b.asok config get fsid 2024-02-22T02:47:19.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-02-22T02:47:19.712 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:47:19.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-02-22T02:47:19.713 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:47:19.714 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-02-22T02:47:19.714 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-02-22T02:47:19.714 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-02-22T02:47:19.714 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:47:19.714 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:47:19.714 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:19.714 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:19.715 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.b.asok 2024-02-22T02:47:19.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:47:19.715 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.b.asok config get mon_host 2024-02-22T02:47:19.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-02-22T02:47:19.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:47:19.825 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:47:19.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-02-22T02:47:19.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:47:19.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-02-22T02:47:19.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-02-22T02:47:20.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:47:20.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:47:20.222 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:47:20.223 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:47:20.223 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:20.223 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:20.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:47:20.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-02-22T02:47:20.434 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:47:20.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-02-22T02:47:20.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:47:20.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-02-22T02:47:20.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:47:20.436 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:47:20.436 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-02-22T02:47:20.436 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-02-22T02:47:20.436 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-02-22T02:47:20.436 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:47:20.436 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:20.436 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:20.436 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.c.asok 2024-02-22T02:47:20.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:47:20.437 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.c.asok config get fsid 2024-02-22T02:47:20.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-02-22T02:47:20.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:47:20.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-02-22T02:47:20.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:47:20.551 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:47:20.552 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-02-22T02:47:20.552 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-02-22T02:47:20.552 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-02-22T02:47:20.552 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:47:20.553 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:20.553 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:20.553 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.c.asok 2024-02-22T02:47:20.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:47:20.553 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.c.asok config get mon_host 2024-02-22T02:47:20.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-02-22T02:47:20.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:47:20.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:47:20.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-02-22T02:47:20.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:47:20.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-02-22T02:47:20.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-02-22T02:47:21.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:47:21.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:47:21.069 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:47:21.069 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:47:21.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:21.070 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:21.070 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:47:21.070 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-02-22T02:47:21.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:47:21.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-02-22T02:47:21.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:47:21.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-02-22T02:47:21.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:47:21.275 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:47:21.276 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-02-22T02:47:21.276 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-02-22T02:47:21.276 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-02-22T02:47:21.277 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:47:21.277 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:21.277 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:21.277 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.d.asok 2024-02-22T02:47:21.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:47:21.277 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.d.asok config get fsid 2024-02-22T02:47:21.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-02-22T02:47:21.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:47:21.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-02-22T02:47:21.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:47:21.395 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:47:21.395 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-02-22T02:47:21.395 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-02-22T02:47:21.395 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-02-22T02:47:21.396 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:47:21.396 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:47:21.396 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:47:21.396 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.d.asok 2024-02-22T02:47:21.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:47:21.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.d.asok config get mon_host 2024-02-22T02:47:21.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2024-02-22T02:47:21.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-02-22T02:47:21.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-02-22T02:47:21.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-02-22T02:47:21.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-02-22T02:47:21.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-02-22T02:47:21.511 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-02-22T02:47:21.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1708570101 2024-02-22T02:47:21.513 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:21.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570041 -lt 1708570101 ]] 2024-02-22T02:47:21.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-02-22T02:47:21.515 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:34.367 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:34.367 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:34.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-02-22T02:47:34.368 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:34.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-02-22T02:47:34.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-02-22T02:47:34.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-02-22T02:47:34.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-02-22T02:47:34.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-02-22T02:47:34.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2024-02-22T02:47:34.768 INFO:tasks.workunit.client.0.smithi046.stderr:invalid mon(s) specified: [dne] 2024-02-22T02:47:34.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-02-22T02:47:35.148 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-02-22T02:47:35.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-02-22T02:47:35.493 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (a,c,d) after stopping [b] 2024-02-22T02:47:35.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-02-22T02:47:35.881 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (a,b,d) after stopping [c] 2024-02-22T02:47:35.897 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-02-22T02:47:36.297 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (a,b,c) after stopping [d] 2024-02-22T02:47:36.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2024-02-22T02:47:36.632 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2024-02-22T02:47:36.635 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2024-02-22T02:47:36.860 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2024-02-22T02:47:36.862 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-02-22T02:47:37.431 INFO:tasks.workunit.client.0.smithi046.stderr:adding another mon that is not yet online will not break quorum 2024-02-22T02:47:37.454 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-02-22T02:47:40.692 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.a 2024-02-22T02:47:40.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-02-22T02:47:41.011 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.b 2024-02-22T02:47:41.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-02-22T02:47:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.c 2024-02-22T02:47:41.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-02-22T02:47:41.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:47:41.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:47:41.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:47:41.411 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:47:41.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:47:41.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:47:41.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2024-02-22T02:47:41.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-02-22T02:47:41.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-02-22T02:47:41.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-02-22T02:47:41.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-02-22T02:47:41.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-02-22T02:47:41.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-02-22T02:47:41.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1708570121 2024-02-22T02:47:41.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:41.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570061 -lt 1708570121 ]] 2024-02-22T02:47:41.521 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:47:41.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:41.760 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:41.760 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:41.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:47:41.761 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:41.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:41.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:41.791 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:41.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570061 -lt 1708570121 ]] 2024-02-22T02:47:41.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:47:41.792 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:42.027 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:42.027 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:42.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:47:42.028 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:42.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:42.065 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:42.065 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:42.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570062 -lt 1708570121 ]] 2024-02-22T02:47:42.066 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:47:42.066 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:42.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:42.299 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:42.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:42.300 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:47:42.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:42.337 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:42.337 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:42.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570062 -lt 1708570121 ]] 2024-02-22T02:47:42.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:47:42.338 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:42.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:42.571 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:42.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:42.572 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:47:42.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:42.611 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:42.611 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:42.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570062 -lt 1708570121 ]] 2024-02-22T02:47:42.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:47:42.612 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:42.845 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:42.845 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:42.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:42.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:47:42.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:42.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:42.876 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:42.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570062 -lt 1708570121 ]] 2024-02-22T02:47:42.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:47:42.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:43.116 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:43.117 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:43.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:47:43.117 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:43.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:43.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:43.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:43.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570063 -lt 1708570121 ]] 2024-02-22T02:47:43.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:47:43.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:46.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:46.396 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:46.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:46.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:47:46.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:46.433 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:46.433 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:46.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570066 -lt 1708570121 ]] 2024-02-22T02:47:46.435 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:47:46.435 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:46.664 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:46.664 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:46.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:47:46.665 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:46.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:46.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:46.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:46.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570066 -lt 1708570121 ]] 2024-02-22T02:47:46.697 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:47:46.697 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:46.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:46.951 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:46.951 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:47:46.952 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:46.981 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:46.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:46.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:46.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570066 -lt 1708570121 ]] 2024-02-22T02:47:46.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:47:46.983 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:50.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:50.225 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:50.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:50.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:47:50.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:50.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:50.261 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:50.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570070 -lt 1708570121 ]] 2024-02-22T02:47:50.262 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:47:50.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:50.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:50.496 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:50.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:47:50.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:50.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:47:50.531 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:47:50.531 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:47:50.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570070 -lt 1708570121 ]] 2024-02-22T02:47:50.533 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:47:50.533 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:47:56.864 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:47:56.864 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:56.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:47:56.865 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:47:18.098193Z","created":"2024-02-22T02:47:18.098193Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:47:56.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-02-22T02:47:56.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-02-22T02:47:56.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-02-22T02:47:56.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-02-22T02:47:56.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-02-22T02:47:56.905 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-02-22T02:47:57.235 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-02-22T02:47:57.251 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-02-22T02:47:57.494 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-02-22T02:47:57.498 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-02-22T02:47:57.821 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-02-22T02:47:57.826 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-02-22T02:47:58.037 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-02-22T02:47:58.040 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-02-22T02:47:58.359 INFO:tasks.workunit.client.0.smithi046.stderr:adding another mon that is not yet online will not break quorum 2024-02-22T02:47:58.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-02-22T02:47:58.730 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.a 2024-02-22T02:47:58.745 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-02-22T02:48:01.974 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.b 2024-02-22T02:48:01.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-02-22T02:48:02.318 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.c 2024-02-22T02:48:02.334 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2024-02-22T02:48:02.678 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.d 2024-02-22T02:48:02.694 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-02-22T02:48:02.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:48:02.695 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:48:02.695 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:48:02.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:48:02.696 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:48:02.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:48:02.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-02-22T02:48:02.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:48:02.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:48:02.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:48:02.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:48:02.805 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:48:02.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:48:02.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:48:02.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:48:02.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:48:02.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:48:02.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:48:02.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:48:02.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:48:02.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:48:02.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:48:02.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:48:02.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:48:02.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:48:02.818 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:48:02.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:48:02.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:48:02.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:48:02.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:48:02.826 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:02.827 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:02.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:48:02.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:48:02.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:48:02.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-02-22T02:48:02.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-02-22T02:48:02.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-02-22T02:48:02.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-02-22T02:48:02.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:48:02.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:48:02.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:48:02.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:48:02.829 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:48:02.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:48:02.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:48:02.829 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:48:02.830 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:48:02.831 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:48:02.831 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:48:02.832 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:48:02.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:48:02.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:48:02.833 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:48:02.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:48:02.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:48:02.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:48:02.835 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:48:02.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:48:02.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:48:02.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:48:02.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:48:02.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:02.838 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:02.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:48:02.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:48:02.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:48:02.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-02-22T02:48:02.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:48:02.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:02.840 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:02.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.46785 2024-02-22T02:48:02.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:48:02.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:48:02.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:48:02.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-02-22T02:48:02.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2024-02-22T02:48:02.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2024-02-22T02:48:02.843 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=1eb0880b-a2ad-4980-becd-4e67c2d63ae9 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2024-02-22T02:48:02.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-02-22T02:48:02.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:48:02.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:48:02.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:48:02.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:48:02.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-02-22T02:48:02.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-02-22T02:48:03.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:48:03.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:48:03.273 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:48:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:48:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:03.273 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:03.274 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:48:03.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-02-22T02:48:03.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:48:03.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:48:03.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:48:03.518 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:48:03.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:48:03.519 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:48:03.520 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:48:03.520 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:48:03.520 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:48:03.520 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:48:03.520 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:03.520 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:03.520 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:48:03.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:48:03.520 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get fsid 2024-02-22T02:48:03.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:48:03.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:48:03.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:48:03.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:48:03.639 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:48:03.639 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:48:03.640 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:48:03.640 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:48:03.641 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:48:03.641 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:03.641 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:03.641 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.a.asok 2024-02-22T02:48:03.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:48:03.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.a.asok config get mon_host 2024-02-22T02:48:03.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-02-22T02:48:03.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:48:03.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:48:03.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-02-22T02:48:03.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:48:03.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-02-22T02:48:03.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-02-22T02:48:04.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:48:04.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:48:04.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:48:04.193 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:48:04.193 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:04.193 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:04.193 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:48:04.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-02-22T02:48:04.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:48:04.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-02-22T02:48:04.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:48:04.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-02-22T02:48:04.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:48:04.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:48:04.406 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-02-22T02:48:04.406 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-02-22T02:48:04.407 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-02-22T02:48:04.407 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:48:04.407 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:04.407 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:04.407 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.b.asok 2024-02-22T02:48:04.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:48:04.407 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.b.asok config get fsid 2024-02-22T02:48:04.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-02-22T02:48:04.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:48:04.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-02-22T02:48:04.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:48:04.525 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:48:04.526 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-02-22T02:48:04.526 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-02-22T02:48:04.526 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-02-22T02:48:04.526 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:48:04.526 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:04.526 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:04.527 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.b.asok 2024-02-22T02:48:04.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:48:04.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.b.asok config get mon_host 2024-02-22T02:48:04.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-02-22T02:48:04.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:48:04.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:48:04.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-02-22T02:48:04.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:48:04.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-02-22T02:48:04.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-02-22T02:48:05.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:48:05.054 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:48:05.055 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:48:05.055 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:48:05.055 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:05.055 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:05.056 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:48:05.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-02-22T02:48:05.283 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:48:05.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-02-22T02:48:05.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:48:05.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-02-22T02:48:05.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:48:05.284 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:48:05.285 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-02-22T02:48:05.285 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-02-22T02:48:05.285 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-02-22T02:48:05.286 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:48:05.286 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:05.286 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:05.286 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.c.asok 2024-02-22T02:48:05.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:48:05.286 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.c.asok config get fsid 2024-02-22T02:48:05.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-02-22T02:48:05.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:48:05.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-02-22T02:48:05.404 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:48:05.406 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-02-22T02:48:05.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:48:05.406 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-02-22T02:48:05.406 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-02-22T02:48:05.406 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:48:05.406 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:05.406 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:05.406 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.c.asok 2024-02-22T02:48:05.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:48:05.406 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.c.asok config get mon_host 2024-02-22T02:48:05.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-02-22T02:48:05.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:48:05.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:48:05.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-02-22T02:48:05.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:48:05.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-02-22T02:48:05.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-02-22T02:48:05.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:48:05.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:48:05.917 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:48:05.918 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:48:05.918 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:05.918 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:05.918 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:48:05.918 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-02-22T02:48:06.122 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:48:06.122 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-02-22T02:48:06.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:48:06.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-02-22T02:48:06.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:48:06.124 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:48:06.124 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-02-22T02:48:06.124 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-02-22T02:48:06.124 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-02-22T02:48:06.124 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:48:06.124 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:06.124 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:06.125 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.d.asok 2024-02-22T02:48:06.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:48:06.125 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.d.asok config get fsid 2024-02-22T02:48:06.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-02-22T02:48:06.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:48:06.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-02-22T02:48:06.242 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:48:06.243 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:48:06.243 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-02-22T02:48:06.243 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-02-22T02:48:06.243 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-02-22T02:48:06.244 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:48:06.244 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:06.244 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:06.244 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.d.asok 2024-02-22T02:48:06.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:48:06.244 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.d.asok config get mon_host 2024-02-22T02:48:06.351 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2024-02-22T02:48:06.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-02-22T02:48:06.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:48:06.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=e 2024-02-22T02:48:06.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:48:06.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/e 2024-02-22T02:48:06.352 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2024-02-22T02:48:06.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:48:06.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:48:06.846 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:48:06.847 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:48:06.847 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:06.847 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:06.847 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46785/$cluster-$name.asok' 2024-02-22T02:48:06.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.46785/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2024-02-22T02:48:07.058 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:48:07.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e fsid 2024-02-22T02:48:07.059 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:48:07.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-02-22T02:48:07.060 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:48:07.061 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:48:07.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-02-22T02:48:07.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-02-22T02:48:07.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-02-22T02:48:07.061 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:48:07.061 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:07.061 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:07.061 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.e.asok 2024-02-22T02:48:07.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:48:07.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.e.asok config get fsid 2024-02-22T02:48:07.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e mon_host 2024-02-22T02:48:07.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:48:07.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-02-22T02:48:07.175 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:48:07.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:48:07.176 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-02-22T02:48:07.176 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-02-22T02:48:07.176 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-02-22T02:48:07.177 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:48:07.177 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:48:07.177 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:48:07.177 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46785/ceph-mon.e.asok 2024-02-22T02:48:07.177 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:48:07.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46785/ceph-mon.e.asok config get mon_host 2024-02-22T02:48:07.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2024-02-22T02:48:07.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-02-22T02:48:07.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=5 2024-02-22T02:48:07.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-02-22T02:48:07.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 5 ]] 2024-02-22T02:48:07.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-02-22T02:48:07.281 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-02-22T02:48:07.282 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1708570147 2024-02-22T02:48:07.283 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:07.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570087 -lt 1708570147 ]] 2024-02-22T02:48:07.285 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-02-22T02:48:07.285 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:19.526 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:19.526 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:19.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 5' 2024-02-22T02:48:19.527 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:19.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-02-22T02:48:19.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-02-22T02:48:19.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-02-22T02:48:19.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-02-22T02:48:19.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-02-22T02:48:19.557 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2024-02-22T02:48:19.906 INFO:tasks.workunit.client.0.smithi046.stderr:invalid mon(s) specified: [dne] 2024-02-22T02:48:19.922 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-02-22T02:48:20.284 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-02-22T02:48:20.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-02-22T02:48:20.620 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2024-02-22T02:48:20.637 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-02-22T02:48:20.964 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2024-02-22T02:48:20.980 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-02-22T02:48:21.344 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2024-02-22T02:48:21.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-02-22T02:48:21.693 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2024-02-22T02:48:21.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2024-02-22T02:48:22.082 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2024-02-22T02:48:22.098 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2024-02-22T02:48:22.499 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2024-02-22T02:48:22.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2024-02-22T02:48:22.830 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2024-02-22T02:48:22.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-02-22T02:48:23.207 INFO:tasks.workunit.client.0.smithi046.stderr:adding another mon that is not yet online will not break quorum 2024-02-22T02:48:23.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-02-22T02:48:23.615 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.a 2024-02-22T02:48:23.631 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-02-22T02:48:24.074 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.b 2024-02-22T02:48:24.090 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-02-22T02:48:24.498 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.c 2024-02-22T02:48:24.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-02-22T02:48:24.884 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.d 2024-02-22T02:48:24.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-02-22T02:48:25.255 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.e 2024-02-22T02:48:25.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-02-22T02:48:25.271 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:48:25.272 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:48:25.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:48:25.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:48:25.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:48:25.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:48:25.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2024-02-22T02:48:25.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-02-22T02:48:25.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-02-22T02:48:25.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-02-22T02:48:25.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-02-22T02:48:25.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-02-22T02:48:25.382 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-02-22T02:48:25.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1708570165 2024-02-22T02:48:25.383 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:25.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570105 -lt 1708570165 ]] 2024-02-22T02:48:25.383 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-02-22T02:48:25.384 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:25.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:25.616 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:25.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:25.617 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-02-22T02:48:25.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:25.654 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:25.654 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:25.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570105 -lt 1708570165 ]] 2024-02-22T02:48:25.655 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-02-22T02:48:25.655 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:25.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:25.893 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:25.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-02-22T02:48:25.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:25.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:25.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:25.930 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:25.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570105 -lt 1708570165 ]] 2024-02-22T02:48:25.931 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-02-22T02:48:25.931 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:29.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:29.182 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:29.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-02-22T02:48:29.182 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:29.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:29.211 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:29.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:29.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570109 -lt 1708570165 ]] 2024-02-22T02:48:29.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-02-22T02:48:29.213 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:29.439 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:29.439 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:29.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-02-22T02:48:29.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:29.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:29.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:29.476 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:29.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570109 -lt 1708570165 ]] 2024-02-22T02:48:29.476 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-02-22T02:48:29.477 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:32.718 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:32.718 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:32.718 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-02-22T02:48:32.719 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:32.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:32.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:32.750 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:32.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570112 -lt 1708570165 ]] 2024-02-22T02:48:32.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-02-22T02:48:32.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:32.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:32.993 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:32.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:32.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-02-22T02:48:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:33.030 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:33.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:33.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570113 -lt 1708570165 ]] 2024-02-22T02:48:33.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-02-22T02:48:33.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:33.267 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:33.267 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:33.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-02-22T02:48:33.268 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:33.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:33.305 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:33.305 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:33.306 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570113 -lt 1708570165 ]] 2024-02-22T02:48:33.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-02-22T02:48:33.307 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:33.536 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:33.536 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:33.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:33.537 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-02-22T02:48:33.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:33.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:33.567 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:33.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570113 -lt 1708570165 ]] 2024-02-22T02:48:33.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-02-22T02:48:33.569 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:33.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:33.814 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:33.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-02-22T02:48:33.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:33.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:33.852 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:33.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:33.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570113 -lt 1708570165 ]] 2024-02-22T02:48:33.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-02-22T02:48:33.853 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:34.086 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:34.086 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:34.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:34.087 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-02-22T02:48:34.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:34.123 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:34.123 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:34.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570114 -lt 1708570165 ]] 2024-02-22T02:48:34.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-02-22T02:48:34.126 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:34.357 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:34.357 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-02-22T02:48:34.358 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:34.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:34.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:34.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:34.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570114 -lt 1708570165 ]] 2024-02-22T02:48:34.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-02-22T02:48:34.394 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:40.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:40.843 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:40.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-02-22T02:48:40.844 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:40.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-02-22T02:48:40.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-02-22T02:48:40.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-02-22T02:48:40.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-02-22T02:48:40.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-02-22T02:48:40.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-02-22T02:48:41.249 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-02-22T02:48:41.265 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-02-22T02:48:41.579 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (c,d,e) after stopping [b] 2024-02-22T02:48:41.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-02-22T02:48:41.951 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (b,d,e) after stopping [c] 2024-02-22T02:48:41.967 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-02-22T02:48:45.199 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (b,c,e) after stopping [d] 2024-02-22T02:48:45.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-02-22T02:48:45.555 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-02-22T02:48:45.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-02-22T02:48:45.870 INFO:tasks.workunit.client.0.smithi046.stderr:adding another mon that is not yet online will not break quorum 2024-02-22T02:48:45.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-02-22T02:48:46.292 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.a 2024-02-22T02:48:46.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-02-22T02:48:46.659 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.b 2024-02-22T02:48:46.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-02-22T02:48:46.992 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.c 2024-02-22T02:48:47.010 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-02-22T02:48:50.241 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.d 2024-02-22T02:48:50.257 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-02-22T02:48:53.488 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.e 2024-02-22T02:48:53.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2024-02-22T02:48:53.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:48:53.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:48:53.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:48:53.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:48:53.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:48:53.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:48:53.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2024-02-22T02:48:53.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-02-22T02:48:53.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-02-22T02:48:53.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-02-22T02:48:53.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-02-22T02:48:53.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-02-22T02:48:53.613 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-02-22T02:48:53.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1708570193 2024-02-22T02:48:53.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:53.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570133 -lt 1708570193 ]] 2024-02-22T02:48:53.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:48:53.616 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:53.854 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:53.854 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:53.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:53.855 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:48:53.892 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:53.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:53.893 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:53.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570133 -lt 1708570193 ]] 2024-02-22T02:48:53.893 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:48:53.894 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:57.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:57.139 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:57.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:48:57.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:57.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:57.176 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:57.176 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:57.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570137 -lt 1708570193 ]] 2024-02-22T02:48:57.178 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:48:57.178 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:48:57.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:48:57.410 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:57.410 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:48:57.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:48:57.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:48:57.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:48:57.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:48:57.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570137 -lt 1708570193 ]] 2024-02-22T02:48:57.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:48:57.445 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:49:00.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:49:00.678 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:49:00.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:49:00.679 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:49:00.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:49:00.709 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:49:00.709 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:49:00.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570140 -lt 1708570193 ]] 2024-02-22T02:49:00.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:49:00.710 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:49:00.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:49:00.946 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:49:00.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:49:00.947 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:49:00.982 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:49:00.983 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:49:00.983 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:49:00.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570140 -lt 1708570193 ]] 2024-02-22T02:49:00.984 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:49:00.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:49:01.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:49:01.218 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:49:01.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:49:01.218 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:49:01.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:49:01.247 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:49:01.247 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:49:01.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570141 -lt 1708570193 ]] 2024-02-22T02:49:01.248 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:49:01.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:49:01.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:49:01.469 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:49:01.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:49:01.470 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:49:01.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-02-22T02:49:01.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-02-22T02:49:01.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-02-22T02:49:01.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1708570141 -lt 1708570193 ]] 2024-02-22T02:49:01.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-02-22T02:49:01.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-02-22T02:49:09.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-02-22T02:49:09.191 INFO:tasks.workunit.client.0.smithi046.stderr:{"election_epoch":12,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:49:09.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1eb0880b-a2ad-4980-becd-4e67c2d63ae9","modified":"2024-02-22T02:48:02.871335Z","created":"2024-02-22T02:48:02.871335Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-02-22T02:49:09.191 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-02-22T02:49:09.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-02-22T02:49:09.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-02-22T02:49:09.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-02-22T02:49:09.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-02-22T02:49:09.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-02-22T02:49:09.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-02-22T02:49:12.450 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-02-22T02:49:12.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-02-22T02:49:15.656 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-02-22T02:49:15.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-02-22T02:49:15.914 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-02-22T02:49:15.919 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-02-22T02:49:16.184 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-02-22T02:49:16.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-02-22T02:49:16.483 INFO:tasks.workunit.client.0.smithi046.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-02-22T02:49:16.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-02-22T02:49:16.806 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-02-22T02:49:16.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-02-22T02:49:17.130 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.a 2024-02-22T02:49:17.147 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-02-22T02:49:17.499 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: removing mon.b would break quorum 2024-02-22T02:49:17.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-02-22T02:49:17.783 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: removing mon.c would break quorum 2024-02-22T02:49:17.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-02-22T02:49:18.068 INFO:tasks.workunit.client.0.smithi046.stderr:Error EBUSY: removing mon.d would break quorum 2024-02-22T02:49:18.072 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-02-22T02:49:21.314 INFO:tasks.workunit.client.0.smithi046.stderr:safe to remove mon.e 2024-02-22T02:49:21.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-02-22T02:49:21.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:49:21.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:49:21.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:49:21.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:49:21.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:49:21.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:49:21.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-02-22T02:49:21.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:49:21.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:49:21.440 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:49:21.440 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:49:21.441 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:49:21.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:49:21.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:49:21.441 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:49:21.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:49:21.448 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:49:21.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:49:21.449 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:49:21.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:49:21.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:49:21.450 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:49:21.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:49:21.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:49:21.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:49:21.451 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:49:21.451 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:49:21.452 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:49:21.453 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:49:21.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:49:21.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:49:21.464 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:49:21.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:49:21.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:49:21.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:49:21.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-22T02:49:21.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/ok-to-stop 0 2024-02-22T02:49:21.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-02-22T02:49:21.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-22T02:49:21.465 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-02-22T02:49:21.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:49:21.466 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:49:21.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:49:21.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:49:21.466 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:49:21.467 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:49:21.467 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:49:21.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:49:21.468 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:49:21.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:49:21.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:49:21.469 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:49:21.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:49:21.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:49:21.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:49:21.471 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:49:21.471 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:49:21.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-22T02:49:21.472 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-02-22T02:49:21.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:49:21.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:49:21.473 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46785 2024-02-22T02:49:21.473 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46785 2024-02-22T02:49:21.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:49:21.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:49:21.475 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-22T02:49:21.475 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-22T02:49:21.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-22T02:49:21.527 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2024-02-22T02:49:21.527 DEBUG:teuthology.orchestra.run.smithi046:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbe7fcec16eddee6239e4dd68c4d0203c7df6461 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2024-02-22T02:49:21.579 INFO:tasks.workunit.client.0.smithi046.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-22T02:49:21.585 INFO:tasks.workunit.client.0.smithi046.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-22T02:49:21.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-22T02:49:21.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-22T02:49:21.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-22T02:49:21.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-22T02:49:21.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-22T02:49:21.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-22T02:49:21.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-22T02:49:21.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-22T02:49:21.585 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/rados-striper 2024-02-22T02:49:21.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2024-02-22T02:49:21.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2024-02-22T02:49:21.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2024-02-22T02:49:21.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2024-02-22T02:49:21.586 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2024-02-22T02:49:21.586 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2024-02-22T02:49:21.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=49220fe0-4f1c-44b2-b6ae-3187170d01cc --auth-supported=none ' 2024-02-22T02:49:21.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2024-02-22T02:49:21.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2024-02-22T02:49:21.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/rados-striper 2024-02-22T02:49:21.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/rados-striper 2024-02-22T02:49:21.587 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-02-22T02:49:21.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:49:21.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-02-22T02:49:21.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:49:21.588 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:49:21.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:49:21.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:49:21.588 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:49:21.589 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:49:21.589 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:49:21.590 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:49:21.590 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:49:21.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:49:21.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:49:21.592 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:49:21.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:49:21.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:49:21.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:49:21.593 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:49:21.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:49:21.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:49:21.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-02-22T02:49:21.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:49:21.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:49:21.596 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63524 2024-02-22T02:49:21.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.63524 2024-02-22T02:49:21.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:49:21.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:49:21.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/rados-striper 2024-02-22T02:49:21.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:49:21.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:49:21.598 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63524 2024-02-22T02:49:21.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.63524 2024-02-22T02:49:21.600 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:49:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-22T02:49:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-22T02:49:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:49:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2024-02-22T02:49:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2024-02-22T02:49:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/rados-striper 2024-02-22T02:49:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:49:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:49:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:49:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/rados-striper/a 2024-02-22T02:49:21.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2024-02-22T02:49:21.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:49:21.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:49:21.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:49:21.995 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:49:21.995 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:49:21.995 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63524 2024-02-22T02:49:21.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.63524/$cluster-$name.asok' 2024-02-22T02:49:21.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.63524/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-22T02:49:22.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:49:22.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:49:22.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:49:22.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:49:22.208 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:49:22.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:49:22.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:49:22.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:49:22.209 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:49:22.209 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:49:22.210 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:49:22.210 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63524 2024-02-22T02:49:22.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.63524/ceph-mon.a.asok 2024-02-22T02:49:22.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:49:22.210 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.63524/ceph-mon.a.asok config get fsid 2024-02-22T02:49:22.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:49:22.327 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:49:22.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:49:22.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:49:22.329 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:49:22.329 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:49:22.330 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:49:22.330 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:49:22.330 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:49:22.330 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:49:22.330 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63524 2024-02-22T02:49:22.330 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.63524/ceph-mon.a.asok 2024-02-22T02:49:22.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:49:22.330 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.63524/ceph-mon.a.asok config get mon_host 2024-02-22T02:49:22.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2024-02-22T02:49:22.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-02-22T02:49:22.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:49:22.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:49:22.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:49:22.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/0 2024-02-22T02:49:22.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=49220fe0-4f1c-44b2-b6ae-3187170d01cc --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-02-22T02:49:22.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:49:22.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:49:22.442 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:49:22.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2024-02-22T02:49:22.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2024-02-22T02:49:22.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:49:22.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:49:22.443 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-02-22T02:49:22.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:49:22.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:49:22.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:49:22.443 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:49:22.443 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:49:22.443 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63524 2024-02-22T02:49:22.443 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.63524/$cluster-$name.asok' 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63524/$cluster-$name.asok' 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/0 2024-02-22T02:49:22.444 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:49:22.445 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=404f8657-e154-4685-aa4c-c1f06a0b65a1 2024-02-22T02:49:22.445 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 404f8657-e154-4685-aa4c-c1f06a0b65a1 2024-02-22T02:49:22.446 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 404f8657-e154-4685-aa4c-c1f06a0b65a1' 2024-02-22T02:49:22.446 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:49:22.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAyttZlB635GxAAPyC1uNKy6CpvhBCtG82FPQ== 2024-02-22T02:49:22.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAyttZlB635GxAAPyC1uNKy6CpvhBCtG82FPQ=="}' 2024-02-22T02:49:22.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 404f8657-e154-4685-aa4c-c1f06a0b65a1 -i td/rados-striper/0/new.json 2024-02-22T02:49:23.367 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:49:23.386 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/0/new.json 2024-02-22T02:49:23.388 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=49220fe0-4f1c-44b2-b6ae-3187170d01cc --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63524/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAyttZlB635GxAAPyC1uNKy6CpvhBCtG82FPQ== --osd-uuid 404f8657-e154-4685-aa4c-c1f06a0b65a1 2024-02-22T02:49:23.422 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:23.425+0000 7fbdb02c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:49:23.427 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:23.429+0000 7fbdb02c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:49:23.429 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:23.429+0000 7fbdb02c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:49:23.429 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:23.429+0000 7fbdb02c77c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-02-22T02:49:23.429 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:23.429+0000 7fbdb02c77c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-02-22T02:49:23.429 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:23.429+0000 7fbdb02c77c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-02-22T02:49:23.484 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:23.485+0000 7fbdb02c77c0 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2024-02-22T02:49:29.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/0/keyring 2024-02-22T02:49:29.863 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:49:29.865 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:49:29.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:49:29.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:49:30.116 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:49:30.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:49:30.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=49220fe0-4f1c-44b2-b6ae-3187170d01cc --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63524/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:49:30.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:49:30.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:49:30.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:49:30.142 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:30.145+0000 7f8775c8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:49:30.144 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:30.145+0000 7f8775c8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:49:30.146 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:30.149+0000 7f8775c8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:49:30.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:49:30.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:49:30.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:49:30.344 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:49:30.344 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:49:30.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:49:30.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:30.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:49:30.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:30.345 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:30.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:30.712 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:30.713+0000 7f8775c8a7c0 -1 Falling back to public interface 2024-02-22T02:49:31.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:31.580 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:31.580 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:49:31.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:49:31.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:31.581 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:31.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:32.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:32.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:32.817 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:49:32.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:49:32.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:32.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:33.053 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:33.431 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:33.433+0000 7f8775c8a7c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:49:34.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:34.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:34.055 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:49:34.055 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:49:34.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:34.056 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:34.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:35.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:35.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:35.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:49:35.514 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:49:35.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:35.514 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:35.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:36.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:36.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:36.752 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:49:36.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:49:36.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:36.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:36.988 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:37.989 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:37.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:37.990 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:49:37.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:49:37.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:37.990 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:38.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:39.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:39.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:39.225 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:49:39.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:49:39.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:39.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:39.460 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:40.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:40.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:40.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:49:40.462 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:49:40.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:40.463 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:40.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:41.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:49:41.705 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:49:41.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:41.705 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:41.941 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:42.942 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:42.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:42.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:49:42.943 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:49:42.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:42.943 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:43.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:44.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:44.181 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:44.182 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:49:44.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:49:44.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:44.182 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:44.416 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:45.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:45.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:45.418 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:49:45.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:49:45.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:45.419 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:45.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:46.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:46.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:46.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:49:46.659 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:49:46.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:46.660 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:46.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:47.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:47.900 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:47.901 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:49:47.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:49:47.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:47.902 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:48.139 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:49.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:49.141 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:49.142 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:49:49.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:49:49.142 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:49.143 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:49.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:50.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:50.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:50.381 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:49:50.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:49:50.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:50.382 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:50.615 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:51.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:51.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:51.617 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:49:51.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:49:51.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:51.618 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:51.853 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:52.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:52.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:52.855 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-22T02:49:52.855 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-02-22T02:49:52.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:52.856 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:53.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:54.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:54.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:54.093 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-02-22T02:49:54.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-22T02:49:54.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:54.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:54.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:55.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:55.330 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:55.330 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-02-22T02:49:55.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-22T02:49:55.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:55.331 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:55.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:49:56.569 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:49:56.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:49:56.570 INFO:tasks.workunit.client.0.smithi046.stdout:21 2024-02-22T02:49:56.570 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-22T02:49:56.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:49:56.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:49:56.801 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1146071429,v1:127.0.0.1:6801/1146071429] [v2:127.0.0.1:6802/1146071429,v1:127.0.0.1:6803/1146071429] exists,up 404f8657-e154-4685-aa4c-c1f06a0b65a1 2024-02-22T02:49:56.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:49:56.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:49:56.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:49:56.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2024-02-22T02:49:56.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-02-22T02:49:56.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:49:56.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-22T02:49:56.805 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/1 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=49220fe0-4f1c-44b2-b6ae-3187170d01cc --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:49:56.806 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63524 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.63524/$cluster-$name.asok' 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63524/$cluster-$name.asok' 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:49:56.807 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/1 2024-02-22T02:49:56.808 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:49:56.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ee2336f-a5f1-48c1-8413-63c68fc4161d 2024-02-22T02:49:56.808 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6ee2336f-a5f1-48c1-8413-63c68fc4161d' 2024-02-22T02:49:56.808 INFO:tasks.workunit.client.0.smithi046.stdout:add osd1 6ee2336f-a5f1-48c1-8413-63c68fc4161d 2024-02-22T02:49:56.809 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:49:56.827 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBUttZl6A+0MRAAEc4cPtKxirvXT9GO3QYy7w== 2024-02-22T02:49:56.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBUttZl6A+0MRAAEc4cPtKxirvXT9GO3QYy7w=="}' 2024-02-22T02:49:56.828 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ee2336f-a5f1-48c1-8413-63c68fc4161d -i td/rados-striper/1/new.json 2024-02-22T02:49:57.184 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:49:57.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/1/new.json 2024-02-22T02:49:57.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=49220fe0-4f1c-44b2-b6ae-3187170d01cc --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63524/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBUttZl6A+0MRAAEc4cPtKxirvXT9GO3QYy7w== --osd-uuid 6ee2336f-a5f1-48c1-8413-63c68fc4161d 2024-02-22T02:49:57.238 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:57.245+0000 7f0a306587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:49:57.311 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:57.317+0000 7f0a306587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:49:57.313 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:57.317+0000 7f0a306587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:49:57.313 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:57.321+0000 7f0a306587c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-02-22T02:49:57.314 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:57.321+0000 7f0a306587c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-02-22T02:49:57.314 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:57.321+0000 7f0a306587c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-02-22T02:49:57.428 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:49:57.433+0000 7f0a306587c0 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2024-02-22T02:50:03.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/1/keyring 2024-02-22T02:50:03.619 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:50:03.621 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd1 key to auth repository 2024-02-22T02:50:03.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-22T02:50:03.621 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:50:03.870 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.1 2024-02-22T02:50:03.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-22T02:50:03.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=49220fe0-4f1c-44b2-b6ae-3187170d01cc --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63524/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:50:03.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:50:03.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:50:03.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:50:03.902 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:03.909+0000 7f9d68f3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:50:03.904 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:03.909+0000 7f9d68f3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:50:03.906 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:03.913+0000 7f9d68f3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:50:04.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-22T02:50:04.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:50:04.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-22T02:50:04.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:50:04.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:50:04.104 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:04.105 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:50:04.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:50:04.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:04.105 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:04.336 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:04.452 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:04.457+0000 7f9d68f3c7c0 -1 Falling back to public interface 2024-02-22T02:50:05.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:05.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:05.338 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:50:05.339 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:50:05.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:05.339 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:05.575 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:06.577 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:06.577 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:50:06.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:06.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:50:06.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:06.578 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:06.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:07.150 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:07.157+0000 7f9d68f3c7c0 -1 osd.1 0 log_to_monitors true 2024-02-22T02:50:07.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:07.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:07.814 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:50:07.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:50:07.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:07.815 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:08.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:09.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:09.255 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:09.255 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:50:09.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:50:09.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:09.256 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:09.493 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:10.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:10.495 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:10.495 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:50:10.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:50:10.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:10.496 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:10.734 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:11.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:11.736 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:11.736 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:50:11.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:50:11.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:11.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:11.975 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:12.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:12.976 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:12.977 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:50:12.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:50:12.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:12.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:13.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:14.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:14.216 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:14.216 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:50:14.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:50:14.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:14.217 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:14.448 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:15.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:15.449 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:15.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:50:15.450 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:50:15.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:15.450 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:15.689 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:16.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:16.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:16.691 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:50:16.691 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:50:16.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:16.692 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:16.927 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:17.929 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:17.930 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:50:17.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:17.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:50:17.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:17.930 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:18.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:19.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:19.166 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:19.167 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:50:19.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:50:19.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:19.168 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:19.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:20.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:20.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:20.403 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:50:20.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:50:20.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:20.404 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:20.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:21.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:21.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:21.640 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:50:21.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:50:21.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:21.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:21.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:22.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:22.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:22.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:50:22.881 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:50:22.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:22.881 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:23.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:24.120 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:50:24.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:50:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:24.121 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:24.357 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:25.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:25.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:25.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:50:25.359 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:50:25.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:25.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:25.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:26.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:26.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:26.600 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-02-22T02:50:26.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-22T02:50:26.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:26.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:26.832 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:27.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:27.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:27.834 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-02-22T02:50:27.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-22T02:50:27.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:27.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:28.071 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:29.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:29.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:29.073 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-02-22T02:50:29.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-22T02:50:29.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:29.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:29.353 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:30.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:30.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:30.355 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-22T02:50:30.355 INFO:tasks.workunit.client.0.smithi046.stdout:21 2024-02-22T02:50:30.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:30.356 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-22T02:50:30.660 INFO:tasks.workunit.client.0.smithi046.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/939573497,v1:127.0.0.1:6809/939573497] [v2:127.0.0.1:6810/939573497,v1:127.0.0.1:6811/939573497] exists,up 6ee2336f-a5f1-48c1-8413-63c68fc4161d 2024-02-22T02:50:30.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:50:30.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:50:30.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:50:30.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2024-02-22T02:50:30.662 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/2 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=49220fe0-4f1c-44b2-b6ae-3187170d01cc --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:50:30.663 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:50:30.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:50:30.664 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:50:30.664 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:50:30.664 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63524 2024-02-22T02:50:30.664 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.63524/$cluster-$name.asok' 2024-02-22T02:50:30.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63524/$cluster-$name.asok' 2024-02-22T02:50:30.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:50:30.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:50:30.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:50:30.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-02-22T02:50:30.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-02-22T02:50:30.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:50:30.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:50:30.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:50:30.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:50:30.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:50:30.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:50:30.666 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/2 2024-02-22T02:50:30.667 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:50:30.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ec97ccad-c66e-43c4-91b7-1d48eff0041c 2024-02-22T02:50:30.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ec97ccad-c66e-43c4-91b7-1d48eff0041c' 2024-02-22T02:50:30.667 INFO:tasks.workunit.client.0.smithi046.stdout:add osd2 ec97ccad-c66e-43c4-91b7-1d48eff0041c 2024-02-22T02:50:30.668 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:50:30.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB2ttZlg75AKRAA8tg0qDrFTYiq6qsbSHv9oQ== 2024-02-22T02:50:30.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB2ttZlg75AKRAA8tg0qDrFTYiq6qsbSHv9oQ=="}' 2024-02-22T02:50:30.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ec97ccad-c66e-43c4-91b7-1d48eff0041c -i td/rados-striper/2/new.json 2024-02-22T02:50:31.016 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:50:31.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/2/new.json 2024-02-22T02:50:31.034 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=49220fe0-4f1c-44b2-b6ae-3187170d01cc --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63524/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB2ttZlg75AKRAA8tg0qDrFTYiq6qsbSHv9oQ== --osd-uuid ec97ccad-c66e-43c4-91b7-1d48eff0041c 2024-02-22T02:50:31.061 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:31.064+0000 7f41dd9ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:50:31.102 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:31.108+0000 7f41dd9ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:50:31.103 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:31.108+0000 7f41dd9ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:50:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:31.108+0000 7f41dd9ed7c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-02-22T02:50:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:31.108+0000 7f41dd9ed7c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-02-22T02:50:31.104 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:31.108+0000 7f41dd9ed7c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-02-22T02:50:31.141 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:31.148+0000 7f41dd9ed7c0 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2024-02-22T02:50:37.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/2/keyring 2024-02-22T02:50:37.591 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:50:37.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-22T02:50:37.592 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd2 key to auth repository 2024-02-22T02:50:37.592 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:50:37.844 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-22T02:50:37.847 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.2 2024-02-22T02:50:37.847 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:50:37.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=49220fe0-4f1c-44b2-b6ae-3187170d01cc --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63524/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:50:37.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:50:37.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:50:37.875 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:37.880+0000 7effac36d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:50:37.879 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:37.884+0000 7effac36d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:50:37.881 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:37.884+0000 7effac36d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:50:38.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-22T02:50:38.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:50:38.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-22T02:50:38.083 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:50:38.084 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:50:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:50:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:50:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:38.085 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:38.307 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:38.444 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:38.448+0000 7effac36d7c0 -1 Falling back to public interface 2024-02-22T02:50:39.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:39.309 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:39.309 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:50:39.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:50:39.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:39.310 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:39.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:40.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:40.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:40.546 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:50:40.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:50:40.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:40.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:40.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:40.823 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:50:40.828+0000 7effac36d7c0 -1 osd.2 0 log_to_monitors true 2024-02-22T02:50:41.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:41.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:41.783 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:50:41.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:50:41.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:41.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:42.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:43.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:43.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:43.126 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:50:43.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:50:43.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:43.127 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:43.358 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:44.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:44.359 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:44.360 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:50:44.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:50:44.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:44.360 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:44.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:45.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:45.597 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:45.597 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:50:45.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:50:45.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:45.598 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:45.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:46.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:46.835 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:46.835 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:50:46.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:50:46.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:46.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:47.073 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:48.074 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:48.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:48.075 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:50:48.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:50:48.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:48.075 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:48.314 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:49.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:49.316 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:49.316 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:50:49.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:50:49.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:49.317 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:49.552 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:50.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:50.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:50.554 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:50:50.554 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:50:50.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:50.555 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:50.791 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:51.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:51.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:51.793 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:50:51.794 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:50:51.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:51.794 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:52.035 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:53.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:53.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:53.036 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:50:53.037 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:50:53.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:53.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:53.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:54.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:54.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:54.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:50:54.274 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:50:54.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:54.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:54.515 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:55.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:55.517 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:55.517 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:50:55.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:50:55.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:55.518 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:55.753 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:56.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:56.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:56.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:50:56.755 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:50:56.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:56.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:56.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:57.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:57.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:57.997 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:50:57.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:50:57.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:57.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:58.232 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:50:59.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:50:59.234 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:50:59.234 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:50:59.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:50:59.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:50:59.235 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:50:59.468 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:00.469 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:00.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:00.470 INFO:tasks.workunit.client.0.smithi046.stdout:18 2024-02-22T02:51:00.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-22T02:51:00.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:00.470 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:51:00.708 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:01.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:01.710 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:01.710 INFO:tasks.workunit.client.0.smithi046.stdout:19 2024-02-22T02:51:01.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-22T02:51:01.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:01.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:51:01.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:02.949 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:02.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:02.950 INFO:tasks.workunit.client.0.smithi046.stdout:20 2024-02-22T02:51:02.950 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-22T02:51:02.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:02.951 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-22T02:51:03.235 INFO:tasks.workunit.client.0.smithi046.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/544491673,v1:127.0.0.1:6817/544491673] [v2:127.0.0.1:6818/544491673,v1:127.0.0.1:6819/544491673] exists,up ec97ccad-c66e-43c4-91b7-1d48eff0041c 2024-02-22T02:51:03.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:51:03.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:51:03.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:51:03.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2024-02-22T02:51:03.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-22T02:51:03.475 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' does not exist 2024-02-22T02:51:03.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-22T02:51:03.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-22T02:51:03.881 INFO:tasks.workunit.client.0.smithi046.stderr:pool 'rbd' already exists 2024-02-22T02:51:03.899 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-22T02:51:04.901 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-22T02:51:07.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2024-02-22T02:51:07.081 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records in 2024-02-22T02:51:07.081 INFO:tasks.workunit.client.0.smithi046.stderr:1+0 records out 2024-02-22T02:51:07.081 INFO:tasks.workunit.client.0.smithi046.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 0.000149959 s, 8.2 MB/s 2024-02-22T02:51:07.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2024-02-22T02:51:07.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2024-02-22T02:51:07.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2024-02-22T02:51:07.858 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2024-02-22T02:51:07.859 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2024-02-22T02:51:07.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2024-02-22T02:51:07.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2024-02-22T02:51:07.903 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2024-02-22T02:51:07.904 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2024-02-22T02:51:07.932 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2024-02-22T02:51:07.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2024-02-22T02:51:08.173 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2024-02-22T02:51:08.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2024-02-22T02:51:08.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2024-02-22T02:51:08.213 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2024-02-22T02:51:08.801 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2024-02-22T02:51:08.802 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2024-02-22T02:51:08.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2024-02-22T02:51:08.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2024-02-22T02:51:08.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2024-02-22T02:51:08.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2024-02-22T02:51:08.868 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2024-02-22T02:51:08.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2024-02-22T02:51:09.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2024-02-22T02:51:09.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-02-22T02:51:09.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2024-02-22T02:51:09.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2024-02-22T02:51:09.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2024-02-22T02:51:09.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2024-02-22T02:51:09.124 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2024-02-22T02:51:09.125 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2024-02-22T02:51:09.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2024-02-22T02:51:09.126 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2024-02-22T02:51:09.153 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2024-02-22T02:51:09.154 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2024-02-22T02:51:09.250 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2024-02-22T02:51:09.252 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2024-02-22T02:51:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2024-02-22T02:51:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2024-02-22T02:51:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/rados-striper 2024-02-22T02:51:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-02-22T02:51:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local 'expected=No data available' 2024-02-22T02:51:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-02-22T02:51:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-02-22T02:51:09.254 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2024-02-22T02:51:09.291 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-02-22T02:51:09.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-02-22T02:51:09.292 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-02-22T02:51:09.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-02-22T02:51:09.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-02-22T02:51:09.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/rados-striper 2024-02-22T02:51:09.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-02-22T02:51:09.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local 'expected=No data available' 2024-02-22T02:51:09.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-02-22T02:51:09.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-02-22T02:51:09.294 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-02-22T02:51:09.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-02-22T02:51:09.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-02-22T02:51:09.322 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-02-22T02:51:09.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-02-22T02:51:09.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2024-02-22T02:51:09.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2024-02-22T02:51:09.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/rados-striper 2024-02-22T02:51:09.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-02-22T02:51:09.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local 'expected=No such file or directory' 2024-02-22T02:51:09.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-02-22T02:51:09.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-02-22T02:51:09.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: rados --pool rbd --striper stat toyfile 2024-02-22T02:51:09.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-02-22T02:51:09.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-02-22T02:51:09.582 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-02-22T02:51:09.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-02-22T02:51:09.583 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2024-02-22T02:51:09.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/rados-striper 2024-02-22T02:51:09.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-02-22T02:51:09.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local 'expected=No such file or directory' 2024-02-22T02:51:09.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-02-22T02:51:09.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-02-22T02:51:09.584 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2024-02-22T02:51:09.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-02-22T02:51:09.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-02-22T02:51:09.612 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-02-22T02:51:09.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-02-22T02:51:09.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2024-02-22T02:51:09.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-02-22T02:51:09.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:51:09.613 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-02-22T02:51:09.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:51:09.614 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:51:09.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:51:09.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:51:09.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:51:09.721 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:51:09.721 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:51:09.722 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:51:09.722 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:51:09.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:51:09.723 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:51:09.724 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:51:09.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:51:09.724 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:51:09.725 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:51:09.725 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:51:09.726 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:51:09.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:51:09.727 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-02-22T02:51:09.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:51:09.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:09.741 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63524 2024-02-22T02:51:09.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.63524 2024-02-22T02:51:09.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:51:09.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:51:09.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-22T02:51:09.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/rados-striper 0 2024-02-22T02:51:09.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-02-22T02:51:09.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-22T02:51:09.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-02-22T02:51:09.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:51:09.743 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:51:09.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:51:09.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:51:09.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:51:09.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:51:09.745 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:51:09.746 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:51:09.746 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:51:09.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:51:09.747 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:51:09.748 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:51:09.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:51:09.748 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:51:09.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:51:09.749 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:51:09.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:51:09.750 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-22T02:51:09.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-02-22T02:51:09.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:51:09.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:09.752 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63524 2024-02-22T02:51:09.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.63524 2024-02-22T02:51:09.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:51:09.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:51:09.754 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-22T02:51:09.754 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-22T02:51:09.754 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-22T02:51:09.808 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2024-02-22T02:51:09.808 DEBUG:teuthology.orchestra.run.smithi046:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbe7fcec16eddee6239e4dd68c4d0203c7df6461 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2024-02-22T02:51:09.860 INFO:tasks.workunit.client.0.smithi046.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-22T02:51:09.865 INFO:tasks.workunit.client.0.smithi046.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-22T02:51:09.865 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: run_tests: export .:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-22T02:51:09.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2328: export: `.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin': not a valid identifier 2024-02-22T02:51:09.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: run_tests: export CEPH_MON=127.0.0.1:7109 2024-02-22T02:51:09.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: run_tests: CEPH_MON=127.0.0.1:7109 2024-02-22T02:51:09.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2331: run_tests: export CEPH_ARGS 2024-02-22T02:51:09.866 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2332: run_tests: uuidgen 2024-02-22T02:51:09.866 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2332: run_tests: CEPH_ARGS+=' --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none ' 2024-02-22T02:51:09.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2333: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2024-02-22T02:51:09.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: run_tests: export CEPH_CONF=/dev/null 2024-02-22T02:51:09.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: run_tests: CEPH_CONF=/dev/null 2024-02-22T02:51:09.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: run_tests: set 2024-02-22T02:51:09.867 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2024-02-22T02:51:09.872 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: run_tests: local 'funcs=test_activate_osd 2024-02-22T02:51:09.872 INFO:tasks.workunit.client.0.smithi046.stderr:test_activate_osd_after_mark_down 2024-02-22T02:51:09.872 INFO:tasks.workunit.client.0.smithi046.stderr:test_activate_osd_skip_benchmark 2024-02-22T02:51:09.872 INFO:tasks.workunit.client.0.smithi046.stderr:test_destroy_osd 2024-02-22T02:51:09.872 INFO:tasks.workunit.client.0.smithi046.stderr:test_display_logs 2024-02-22T02:51:09.872 INFO:tasks.workunit.client.0.smithi046.stderr:test_erasure_code_plugin_exists 2024-02-22T02:51:09.872 INFO:tasks.workunit.client.0.smithi046.stderr:test_expect_failure 2024-02-22T02:51:09.872 INFO:tasks.workunit.client.0.smithi046.stderr:test_flush_pg_stats 2024-02-22T02:51:09.872 INFO:tasks.workunit.client.0.smithi046.stderr:test_get_config 2024-02-22T02:51:09.872 INFO:tasks.workunit.client.0.smithi046.stderr:test_get_is_making_recovery_progress 2024-02-22T02:51:09.872 INFO:tasks.workunit.client.0.smithi046.stderr:test_get_last_scrub_stamp 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_get_not_primary 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_get_num_active_clean 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_get_num_active_or_peered 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_get_num_pgs 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_get_op_scheduler 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_get_osd_id_used_by_pgs 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_get_osds 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_get_pg 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_get_primary 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_get_timeout_delays 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_is_clean 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_kill_daemon 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_kill_daemons 2024-02-22T02:51:09.873 INFO:tasks.workunit.client.0.smithi046.stderr:test_objectstore_tool 2024-02-22T02:51:09.874 INFO:tasks.workunit.client.0.smithi046.stderr:test_pg_schedule_scrub 2024-02-22T02:51:09.874 INFO:tasks.workunit.client.0.smithi046.stderr:test_pg_scrub 2024-02-22T02:51:09.874 INFO:tasks.workunit.client.0.smithi046.stderr:test_repair 2024-02-22T02:51:09.874 INFO:tasks.workunit.client.0.smithi046.stderr:test_run_in_background 2024-02-22T02:51:09.874 INFO:tasks.workunit.client.0.smithi046.stderr:test_run_mon 2024-02-22T02:51:09.874 INFO:tasks.workunit.client.0.smithi046.stderr:test_run_osd 2024-02-22T02:51:09.874 INFO:tasks.workunit.client.0.smithi046.stderr:test_set_config 2024-02-22T02:51:09.874 INFO:tasks.workunit.client.0.smithi046.stderr:test_setup 2024-02-22T02:51:09.874 INFO:tasks.workunit.client.0.smithi046.stderr:test_teardown 2024-02-22T02:51:09.874 INFO:tasks.workunit.client.0.smithi046.stderr:test_wait_background 2024-02-22T02:51:09.874 INFO:tasks.workunit.client.0.smithi046.stderr:test_wait_for_clean 2024-02-22T02:51:09.874 INFO:tasks.workunit.client.0.smithi046.stderr:test_wait_for_health_ok 2024-02-22T02:51:09.874 INFO:tasks.workunit.client.0.smithi046.stderr:test_wait_for_osd 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr:test_wait_for_peered 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr:test_wait_for_scrub 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr:test_wait_osd_id_used_by_pgs' 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2337: run_tests: local dir=td/ceph-helpers 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_activate_osd td/ceph-helpers 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local dir=td/ceph-helpers 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:888: test_activate_osd: setup td/ceph-helpers 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:51:09.875 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:51:09.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:51:09.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:51:09.876 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:51:09.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:51:09.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:51:09.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:51:09.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:51:09.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:51:09.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:51:09.877 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:51:09.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:51:09.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:51:09.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:51:09.878 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:51:09.879 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:51:09.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:51:09.880 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-02-22T02:51:09.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:51:09.881 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:09.882 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:09.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69653 2024-02-22T02:51:09.882 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:51:09.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:51:09.883 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-02-22T02:51:09.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:51:09.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:09.884 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:09.884 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69653 2024-02-22T02:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-22T02:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-22T02:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-02-22T02:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:890: test_activate_osd: run_mon td/ceph-helpers a 2024-02-22T02:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-02-22T02:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:51:09.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-02-22T02:51:09.888 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-02-22T02:51:10.291 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:51:10.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:51:10.292 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:51:10.292 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:51:10.293 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:10.293 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:10.293 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:51:10.293 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-22T02:51:10.520 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:51:10.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:51:10.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:51:10.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:51:10.521 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:51:10.522 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:51:10.522 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:51:10.522 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:51:10.522 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:51:10.523 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:51:10.523 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:10.523 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:10.523 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-mon.a.asok 2024-02-22T02:51:10.523 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:51:10.523 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69653/ceph-mon.a.asok config get fsid 2024-02-22T02:51:10.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:51:10.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:51:10.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:51:10.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:51:10.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:51:10.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:51:10.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:51:10.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:51:10.642 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:51:10.643 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:10.643 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:10.643 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-mon.a.asok 2024-02-22T02:51:10.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:51:10.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69653/ceph-mon.a.asok config get mon_host 2024-02-22T02:51:10.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: run_mgr td/ceph-helpers x 2024-02-22T02:51:10.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-02-22T02:51:10.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:51:10.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:51:10.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:51:10.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-02-22T02:51:10.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:51:11.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:51:11.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:51:11.682 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:51:11.682 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:51:11.682 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:11.682 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:11.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:51:11.683 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:51:11.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:51:11.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: run_osd td/ceph-helpers 0 2024-02-22T02:51:11.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-02-22T02:51:11.811 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:51:11.812 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:51:11.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:51:11.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:51:11.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-02-22T02:51:11.814 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:51:11.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=39cab1f9-8385-4361-92f6-0cd92b674a53 2024-02-22T02:51:11.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 39cab1f9-8385-4361-92f6-0cd92b674a53' 2024-02-22T02:51:11.815 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 39cab1f9-8385-4361-92f6-0cd92b674a53 2024-02-22T02:51:11.815 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:51:11.833 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCfttZljDEVMhAAppQ2N5SQQ9j5EFLhgGl3HA== 2024-02-22T02:51:11.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCfttZljDEVMhAAppQ2N5SQQ9j5EFLhgGl3HA=="}' 2024-02-22T02:51:11.834 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 39cab1f9-8385-4361-92f6-0cd92b674a53 -i td/ceph-helpers/0/new.json 2024-02-22T02:51:12.131 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:51:12.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-02-22T02:51:12.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfttZljDEVMhAAppQ2N5SQQ9j5EFLhgGl3HA== --osd-uuid 39cab1f9-8385-4361-92f6-0cd92b674a53 2024-02-22T02:51:12.185 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:12.192+0000 7fae163837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:51:12.200 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:12.204+0000 7fae163837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:51:12.201 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:12.208+0000 7fae163837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:51:12.202 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:12.208+0000 7fae163837c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-02-22T02:51:12.202 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:12.208+0000 7fae163837c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-02-22T02:51:12.202 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:12.208+0000 7fae163837c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-02-22T02:51:12.240 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:12.244+0000 7fae163837c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-02-22T02:51:18.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-02-22T02:51:18.698 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:51:18.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:51:18.699 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:51:18.699 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:51:19.273 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:51:19.273 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:51:19.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:51:19.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:51:19.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:51:19.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:51:19.305 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:19.308+0000 7f2ed742b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:51:19.307 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:19.312+0000 7f2ed742b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:51:19.308 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:19.312+0000 7f2ed742b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:51:19.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:51:19.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:51:19.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:51:19.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:51:19.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:51:19.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:19.668 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:51:19.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:51:19.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:19.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:19.839 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:19.844+0000 7f2ed742b7c0 -1 Falling back to public interface 2024-02-22T02:51:20.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:21.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:21.118 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:21.119 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:51:21.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:51:21.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:21.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:21.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:22.529 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:22.532+0000 7f2ed742b7c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:51:22.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:22.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:22.549 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:51:22.550 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:51:22.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:22.550 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:23.278 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:24.279 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:24.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:24.280 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:51:24.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:51:24.280 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:24.281 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:24.779 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:25.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:25.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:25.781 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:51:25.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:51:25.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:25.781 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:26.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:27.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:27.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:27.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:51:27.203 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:51:27.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:27.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:27.741 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:28.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:28.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:28.743 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:51:28.744 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:51:28.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:28.744 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:29.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:30.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:30.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:30.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:51:30.221 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:51:30.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:30.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:30.675 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:31.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:31.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:31.677 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:51:31.677 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:51:31.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:31.678 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:32.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:33.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:33.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:33.149 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:51:33.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:51:33.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:33.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:33.702 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:34.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:34.703 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:34.704 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:51:34.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:51:34.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:34.704 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:35.177 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:36.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:36.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:36.179 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:51:36.179 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:51:36.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:36.180 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:37.274 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:38.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:38.275 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:38.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:51:38.276 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:51:38.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:38.276 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:39.023 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:40.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:40.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:40.025 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:51:40.025 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:51:40.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:40.026 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:40.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:41.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:41.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:41.841 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:51:41.842 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:51:41.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:41.842 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:42.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:43.739 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:43.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:43.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:51:43.740 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:51:43.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:43.740 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:44.836 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:45.838 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:45.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:45.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:51:45.839 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:51:45.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:45.839 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:46.749 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:47.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:47.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:47.751 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:51:47.751 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:51:47.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:47.752 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:48.213 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4230668319,v1:127.0.0.1:6803/4230668319] [v2:127.0.0.1:6804/4230668319,v1:127.0.0.1:6805/4230668319] exists,up 39cab1f9-8385-4361-92f6-0cd92b674a53 2024-02-22T02:51:48.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:51:48.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:51:48.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:51:48.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: get_asok_path osd.0 2024-02-22T02:51:48.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:51:48.215 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:51:48.215 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:51:48.215 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:48.215 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:48.216 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-osd.0.asok 2024-02-22T02:51:48.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: CEPH_ARGS= 2024-02-22T02:51:48.216 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.69653/ceph-osd.0.asok config get osd_max_backfills 2024-02-22T02:51:48.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-02-22T02:51:48.326 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2024-02-22T02:51:48.327 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: grep --quiet osd_max_backfills 2024-02-22T02:51:48.328 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:898: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2024-02-22T02:51:48.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:51:48.328 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:51:48.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:51:48.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:51:48.329 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:51:48.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:51:48.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-02-22T02:51:48.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-02-22T02:51:48.640 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-22T02:51:48.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-22T02:51:48.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-22T02:51:48.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-02-22T02:51:48.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-02-22T02:51:48.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:51:48.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:51:48.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:51:48.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-02-22T02:51:48.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-02-22T02:51:48.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-22T02:51:48.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-22T02:51:48.641 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-02-22T02:51:48.641 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-22T02:51:48.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:51:48.642 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:51:48.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:51:48.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:48.642 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:48.643 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:51:48.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:51:48.643 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:51:48.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-02-22T02:51:48.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-02-22T02:51:48.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:51:48.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:51:48.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:51:48.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:51:48.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-22T02:51:48.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-02-22T02:51:48.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-02-22T02:51:48.647 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:51:48.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-22T02:51:48.648 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-02-22T02:51:48.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-02-22T02:51:48.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-22T02:51:48.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-22T02:51:48.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-22T02:51:48.649 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-22T02:51:48.679 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:48.683+0000 7fea58f7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:51:48.682 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:48.687+0000 7fea58f7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:51:48.684 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:48.687+0000 7fea58f7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:51:49.108 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-22T02:51:49.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:51:49.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:51:49.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:51:49.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:51:49.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:49.109 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:51:49.109 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:51:49.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:49.110 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:49.234 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:49.239+0000 7fea58f7a7c0 -1 Falling back to public interface 2024-02-22T02:51:49.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:50.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:50.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:50.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:51:50.538 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:51:50.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:50.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:51.031 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:52.032 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:52.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:52.033 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:51:52.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:51:52.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:52.033 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:52.219 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:52.223+0000 7fea58f7a7c0 -1 osd.0 6 log_to_monitors true 2024-02-22T02:51:52.510 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:51:53.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:51:53.512 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:51:53.512 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:51:53.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:51:53.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:51:53.513 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:51:53.936 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/1558562684,v1:127.0.0.1:6803/1558562684] [v2:127.0.0.1:6804/1558562684,v1:127.0.0.1:6805/1558562684] exists,up 39cab1f9-8385-4361-92f6-0cd92b674a53 2024-02-22T02:51:53.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:51:53.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:51:53.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:51:53.938 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: get_asok_path osd.0 2024-02-22T02:51:53.938 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:51:53.938 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:51:53.938 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:51:53.938 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:53.938 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:53.938 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-osd.0.asok 2024-02-22T02:51:53.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: CEPH_ARGS= 2024-02-22T02:51:53.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.69653/ceph-osd.0.asok config get osd_max_backfills 2024-02-22T02:51:54.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-02-22T02:51:54.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:903: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-02-22T02:51:54.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd: teardown td/ceph-helpers 2024-02-22T02:51:54.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-02-22T02:51:54.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:51:54.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-02-22T02:51:54.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:51:54.062 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:51:54.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:51:54.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:51:54.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:51:54.372 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:51:54.373 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:51:54.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:51:54.374 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:51:54.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:51:54.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:51:54.379 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:51:54.379 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:51:54.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:51:54.380 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:51:54.380 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:51:54.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:51:54.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:51:54.381 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-02-22T02:51:54.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:51:54.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:54.389 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:54.390 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69653 2024-02-22T02:51:54.391 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:51:54.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:51:54.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-02-22T02:51:54.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2024-02-22T02:51:54.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2024-02-22T02:51:54.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:911: test_activate_osd_after_mark_down: setup td/ceph-helpers 2024-02-22T02:51:54.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-02-22T02:51:54.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-02-22T02:51:54.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-02-22T02:51:54.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:51:54.392 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-02-22T02:51:54.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:51:54.393 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:51:54.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:51:54.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:51:54.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:51:54.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:51:54.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:51:54.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:51:54.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:51:54.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:51:54.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:51:54.399 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:51:54.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:51:54.400 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:51:54.401 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:51:54.401 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:51:54.402 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:51:54.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:51:54.403 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-02-22T02:51:54.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:51:54.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:54.405 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:54.405 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69653 2024-02-22T02:51:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:51:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:51:54.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-02-22T02:51:54.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:51:54.408 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:54.409 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:54.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69653 2024-02-22T02:51:54.411 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:51:54.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:51:54.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:51:54.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-02-22T02:51:54.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2024-02-22T02:51:54.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-02-22T02:51:54.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:51:54.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:51:54.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:51:54.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-02-22T02:51:54.412 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-02-22T02:51:54.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:51:54.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:51:54.816 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:51:54.817 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:51:54.817 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:54.817 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:54.817 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:51:54.818 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-22T02:51:55.029 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:51:55.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:51:55.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:51:55.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:51:55.030 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:51:55.031 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:51:55.031 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:51:55.031 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:51:55.031 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:51:55.032 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:51:55.032 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:55.032 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:55.032 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-mon.a.asok 2024-02-22T02:51:55.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:51:55.032 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69653/ceph-mon.a.asok config get fsid 2024-02-22T02:51:55.148 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:51:55.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:51:55.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:51:55.149 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:51:55.150 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:51:55.150 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:51:55.150 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:51:55.150 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:51:55.150 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:51:55.151 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:55.151 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:55.151 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-mon.a.asok 2024-02-22T02:51:55.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:51:55.151 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69653/ceph-mon.a.asok config get mon_host 2024-02-22T02:51:55.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2024-02-22T02:51:55.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-02-22T02:51:55.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:51:55.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:51:55.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:51:55.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-02-22T02:51:55.261 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:51:56.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:51:56.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:51:56.183 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:51:56.184 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:51:56.184 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:56.184 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:56.184 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:51:56.185 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:51:56.186 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:51:56.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2024-02-22T02:51:56.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-02-22T02:51:56.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:51:56.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:51:56.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:51:56.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-02-22T02:51:56.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-02-22T02:51:56.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:51:56.222 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:51:56.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:51:56.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-02-22T02:51:56.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-02-22T02:51:56.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:51:56.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:51:56.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-02-22T02:51:56.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:51:56.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:51:56.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:51:56.223 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:51:56.223 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:51:56.223 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:51:56.223 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:51:56.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:51:56.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:51:56.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:51:56.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:51:56.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-02-22T02:51:56.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-02-22T02:51:56.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:51:56.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:51:56.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:51:56.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:51:56.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:51:56.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:51:56.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-02-22T02:51:56.226 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:51:56.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=804c83db-c982-47de-940f-2d5acad111a0 2024-02-22T02:51:56.226 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 804c83db-c982-47de-940f-2d5acad111a0 2024-02-22T02:51:56.226 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 804c83db-c982-47de-940f-2d5acad111a0' 2024-02-22T02:51:56.227 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:51:56.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMttZl9Q/wDhAABL8SvFGgWzqxQKd2OKLwjQ== 2024-02-22T02:51:56.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMttZl9Q/wDhAABL8SvFGgWzqxQKd2OKLwjQ=="}' 2024-02-22T02:51:56.243 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 804c83db-c982-47de-940f-2d5acad111a0 -i td/ceph-helpers/0/new.json 2024-02-22T02:51:56.556 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:51:56.571 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-02-22T02:51:56.572 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMttZl9Q/wDhAABL8SvFGgWzqxQKd2OKLwjQ== --osd-uuid 804c83db-c982-47de-940f-2d5acad111a0 2024-02-22T02:51:56.612 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:56.615+0000 7f67b11c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:51:56.642 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:56.647+0000 7f67b11c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:51:56.644 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:56.647+0000 7f67b11c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:51:56.644 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:56.647+0000 7f67b11c27c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-02-22T02:51:56.645 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:56.647+0000 7f67b11c27c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-02-22T02:51:56.645 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:56.647+0000 7f67b11c27c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-02-22T02:51:56.681 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:51:56.683+0000 7f67b11c27c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-02-22T02:52:03.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-02-22T02:52:03.195 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:52:03.197 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:52:03.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:52:03.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:52:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:52:03.773 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:52:03.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:52:03.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:52:03.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:52:03.774 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:52:03.798 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:03.803+0000 7fe3e10587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:03.800 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:03.803+0000 7fe3e10587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:03.801 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:03.807+0000 7fe3e10587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:04.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:52:04.223 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:52:04.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:52:04.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:52:04.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:52:04.224 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:04.224 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:52:04.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:52:04.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:04.225 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:04.378 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:04.383+0000 7fe3e10587c0 -1 Falling back to public interface 2024-02-22T02:52:04.656 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:05.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:05.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:05.658 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:52:05.658 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:52:05.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:05.659 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:06.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:06.968 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:06.971+0000 7fe3e10587c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:52:07.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:07.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:07.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:52:07.130 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:52:07.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:07.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:07.684 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:08.685 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:08.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:08.686 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:52:08.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:52:08.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:08.686 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:09.117 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:10.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:10.119 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:10.119 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:52:10.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:52:10.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:10.120 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:10.535 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:11.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:11.537 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:11.537 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:52:11.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:52:11.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:11.538 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:12.172 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:13.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:13.174 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:13.174 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:52:13.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:52:13.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:13.175 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:13.601 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:14.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:14.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:14.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:52:14.603 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:52:14.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:14.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:15.161 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:16.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:16.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:16.163 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:52:16.163 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:52:16.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:16.164 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:16.614 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:17.616 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:17.617 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:52:17.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:52:17.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:17.617 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:18.128 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:19.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:19.130 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:19.131 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:52:19.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:52:19.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:19.131 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:19.593 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:20.594 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:20.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:20.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:52:20.595 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:52:20.595 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:20.596 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:21.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:22.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:22.409 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:22.410 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:52:22.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:52:22.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:22.410 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:23.197 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:24.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:24.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:24.199 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:52:24.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:52:24.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:24.199 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:25.015 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:26.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:26.017 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:26.018 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:52:26.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:52:26.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:26.019 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:26.921 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:27.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:27.923 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:27.923 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:52:27.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:52:27.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:27.924 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:29.043 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:30.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:30.045 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:30.045 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:52:30.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:52:30.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:30.046 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:30.999 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:32.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:32.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:32.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:52:32.003 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:52:32.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:32.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:32.477 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4165855046,v1:127.0.0.1:6803/4165855046] [v2:127.0.0.1:6804/4165855046,v1:127.0.0.1:6805/4165855046] exists,up 804c83db-c982-47de-940f-2d5acad111a0 2024-02-22T02:52:32.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:52:32.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:52:32.478 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:52:32.478 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-02-22T02:52:32.478 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:52:32.478 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:52:32.480 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:52:32.480 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:32.480 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:32.480 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-osd.0.asok 2024-02-22T02:52:32.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-02-22T02:52:32.480 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.69653/ceph-osd.0.asok config get osd_max_backfills 2024-02-22T02:52:32.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2024-02-22T02:52:32.599 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2024-02-22T02:52:32.600 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2024-02-22T02:52:32.602 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2024-02-22T02:52:32.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:52:32.603 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:52:32.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:52:32.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:52:32.603 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:52:32.912 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:52:32.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: test_activate_osd_after_mark_down: ceph osd down 0 2024-02-22T02:52:33.467 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 is already down. 2024-02-22T02:52:33.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: test_activate_osd_after_mark_down: wait_for_osd down 0 2024-02-22T02:52:33.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-02-22T02:52:33.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:52:33.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:52:33.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:52:33.485 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:33.486 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:52:33.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:52:33.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:33.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-02-22T02:52:33.930 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/4165855046,v1:127.0.0.1:6803/4165855046] [v2:127.0.0.1:6804/4165855046,v1:127.0.0.1:6805/4165855046] exists 804c83db-c982-47de-940f-2d5acad111a0 2024-02-22T02:52:33.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:52:33.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:52:33.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:52:33.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-02-22T02:52:33.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-02-22T02:52:33.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-22T02:52:33.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-22T02:52:33.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-22T02:52:33.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-02-22T02:52:33.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-02-22T02:52:33.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:52:33.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:52:33.934 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:52:33.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:52:33.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-02-22T02:52:33.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-02-22T02:52:33.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:52:33.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:52:33.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:52:33.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:52:33.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-22T02:52:33.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-02-22T02:52:33.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-02-22T02:52:33.936 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:52:33.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-22T02:52:33.937 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-02-22T02:52:33.938 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-02-22T02:52:33.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-22T02:52:33.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-22T02:52:33.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-22T02:52:33.938 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-22T02:52:33.961 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:33.967+0000 7f84605767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:33.964 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:33.967+0000 7f84605767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:33.965 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:33.971+0000 7f84605767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:34.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-22T02:52:34.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:52:34.369 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:52:34.370 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:52:34.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:52:34.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:52:34.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:34.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:52:34.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:34.370 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:34.518 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:34.523+0000 7f84605767c0 -1 Falling back to public interface 2024-02-22T02:52:34.787 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:35.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:35.789 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:35.789 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:52:35.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:52:35.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:35.790 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:36.270 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:37.271 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:37.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:37.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:52:37.272 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:52:37.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:37.272 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:37.720 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:37.723+0000 7f84605767c0 -1 osd.0 6 log_to_monitors true 2024-02-22T02:52:37.810 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:38.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:38.812 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:38.812 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:52:38.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:52:38.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:38.813 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:39.245 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/3436674773,v1:127.0.0.1:6803/3436674773] [v2:127.0.0.1:6804/3436674773,v1:127.0.0.1:6805/3436674773] exists,up 804c83db-c982-47de-940f-2d5acad111a0 2024-02-22T02:52:39.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:52:39.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:52:39.246 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:52:39.247 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-02-22T02:52:39.247 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:52:39.247 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:52:39.249 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:52:39.249 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:39.249 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:39.249 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-osd.0.asok 2024-02-22T02:52:39.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-02-22T02:52:39.249 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.69653/ceph-osd.0.asok config get osd_max_backfills 2024-02-22T02:52:39.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2024-02-22T02:52:39.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-02-22T02:52:39.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2024-02-22T02:52:39.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-02-22T02:52:39.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:52:39.376 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-02-22T02:52:39.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:52:39.377 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:52:39.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:52:39.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:52:39.377 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:52:39.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:52:39.486 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:52:39.486 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:52:39.487 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:52:39.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:52:39.488 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:52:39.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:52:39.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:52:39.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:52:39.489 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:52:39.489 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:52:39.490 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:52:39.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:52:39.491 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-02-22T02:52:39.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:52:39.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:39.497 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:39.497 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69653 2024-02-22T02:52:39.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:52:39.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:52:39.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-02-22T02:52:39.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2024-02-22T02:52:39.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:934: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2024-02-22T02:52:39.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2024-02-22T02:52:39.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-02-22T02:52:39.499 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-02-22T02:52:39.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-02-22T02:52:39.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:52:39.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-02-22T02:52:39.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:52:39.500 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:52:39.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:52:39.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:52:39.500 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:52:39.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:52:39.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:52:39.501 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:52:39.501 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:52:39.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:52:39.502 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:52:39.503 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:52:39.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:52:39.503 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:52:39.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:52:39.504 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:52:39.504 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:52:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:52:39.505 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-02-22T02:52:39.506 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:52:39.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:39.507 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:39.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69653 2024-02-22T02:52:39.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:52:39.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:52:39.507 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-02-22T02:52:39.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:52:39.508 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:39.509 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:39.509 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69653 2024-02-22T02:52:39.510 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:52:39.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:52:39.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:52:39.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-02-22T02:52:39.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2024-02-22T02:52:39.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-02-22T02:52:39.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:52:39.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:52:39.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:52:39.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-02-22T02:52:39.511 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-02-22T02:52:39.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:52:39.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:52:39.934 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:52:39.935 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:52:39.935 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:39.935 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:39.935 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:52:39.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-22T02:52:40.135 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:52:40.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:52:40.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:52:40.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:52:40.136 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:52:40.137 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:52:40.137 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:52:40.137 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:52:40.137 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:52:40.138 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:52:40.138 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:40.138 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:40.138 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-mon.a.asok 2024-02-22T02:52:40.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:52:40.139 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69653/ceph-mon.a.asok config get fsid 2024-02-22T02:52:40.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:52:40.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:52:40.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:52:40.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:52:40.262 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:52:40.262 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:52:40.262 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:52:40.262 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:52:40.263 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:52:40.263 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:40.263 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:40.263 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-mon.a.asok 2024-02-22T02:52:40.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:52:40.263 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69653/ceph-mon.a.asok config get mon_host 2024-02-22T02:52:40.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:939: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2024-02-22T02:52:40.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-02-22T02:52:40.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:52:40.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:52:40.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:52:40.374 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-02-22T02:52:40.375 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:52:41.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:52:41.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:52:41.321 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:52:41.322 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:52:41.322 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:41.322 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:41.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:52:41.322 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:52:41.323 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:52:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-02-22T02:52:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-02-22T02:52:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:52:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:52:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:52:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-02-22T02:52:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-02-22T02:52:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:52:41.393 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:52:41.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:52:41.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-02-22T02:52:41.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-02-22T02:52:41.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:52:41.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:52:41.394 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-02-22T02:52:41.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:52:41.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:52:41.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:52:41.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:52:41.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:52:41.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-02-22T02:52:41.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-02-22T02:52:41.397 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:52:41.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0d70027c-0752-4ea2-9563-33bff9d3c69c 2024-02-22T02:52:41.398 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 0d70027c-0752-4ea2-9563-33bff9d3c69c 2024-02-22T02:52:41.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0d70027c-0752-4ea2-9563-33bff9d3c69c' 2024-02-22T02:52:41.398 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:52:41.417 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5ttZlqSROGRAAbZiOXaPrrXXFeWvhsaUxuw== 2024-02-22T02:52:41.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5ttZlqSROGRAAbZiOXaPrrXXFeWvhsaUxuw=="}' 2024-02-22T02:52:41.418 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0d70027c-0752-4ea2-9563-33bff9d3c69c -i td/ceph-helpers/0/new.json 2024-02-22T02:52:41.757 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:52:41.776 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-02-22T02:52:41.777 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQD5ttZlqSROGRAAbZiOXaPrrXXFeWvhsaUxuw== --osd-uuid 0d70027c-0752-4ea2-9563-33bff9d3c69c 2024-02-22T02:52:41.814 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:41.819+0000 7fa7e59757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:41.834 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:41.839+0000 7fa7e59757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:41.835 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:41.839+0000 7fa7e59757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:41.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:41.839+0000 7fa7e59757c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-02-22T02:52:41.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:41.839+0000 7fa7e59757c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-02-22T02:52:41.836 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:41.839+0000 7fa7e59757c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-02-22T02:52:41.874 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:41.879+0000 7fa7e59757c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-02-22T02:52:48.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-02-22T02:52:48.297 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:52:48.299 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:52:48.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:52:48.299 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:52:48.876 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:52:48.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:52:48.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-02-22T02:52:48.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:52:48.877 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:52:48.878 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:52:48.911 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:48.915+0000 7fee169cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:48.931 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:48.935+0000 7fee169cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:48.932 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:48.935+0000 7fee169cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:49.300 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:52:49.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:52:49.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:52:49.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:52:49.301 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:52:49.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:52:49.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:49.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:52:49.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:49.302 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:49.499 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:49.503+0000 7fee169cb7c0 -1 Falling back to public interface 2024-02-22T02:52:49.755 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:50.756 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:50.757 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:52:50.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:50.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:52:50.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:50.757 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:51.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:52.115 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:52.119+0000 7fee169cb7c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:52:52.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:52.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:52.189 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:52:52.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:52:52.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:52.189 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:52.711 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:53.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:53.713 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:53.713 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:52:53.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:52:53.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:53.714 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:54.207 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/849741005,v1:127.0.0.1:6803/849741005] [v2:127.0.0.1:6804/849741005,v1:127.0.0.1:6805/849741005] exists,up 0d70027c-0752-4ea2-9563-33bff9d3c69c 2024-02-22T02:52:54.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:52:54.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:52:54.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:52:54.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-02-22T02:52:54.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:52:54.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:52:54.210 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:52:54.210 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:54.210 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:54.210 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-osd.0.asok 2024-02-22T02:52:54.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-02-22T02:52:54.211 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.69653/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-02-22T02:52:54.332 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-02-22T02:52:54.334 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-02-22T02:52:54.334 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:52:54.334 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:52:54.334 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:52:54.335 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:54.335 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:54.335 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-osd.0.asok 2024-02-22T02:52:54.335 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-02-22T02:52:54.336 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.69653/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-02-22T02:52:54.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-02-22T02:52:54.462 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2024-02-22T02:52:54.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:52:54.463 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:52:54.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:52:54.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:52:54.464 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:52:54.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:52:54.773 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:950: test_activate_osd_skip_benchmark: ceph osd down 0 2024-02-22T02:52:55.211 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 is already down. 2024-02-22T02:52:55.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: wait_for_osd down 0 2024-02-22T02:52:55.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-02-22T02:52:55.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:52:55.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:52:55.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:52:55.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:55.229 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:52:55.229 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:52:55.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:55.230 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-02-22T02:52:55.641 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/849741005,v1:127.0.0.1:6803/849741005] [v2:127.0.0.1:6804/849741005,v1:127.0.0.1:6805/849741005] exists 0d70027c-0752-4ea2-9563-33bff9d3c69c 2024-02-22T02:52:55.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:52:55.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:52:55.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:52:55.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:955: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-02-22T02:52:55.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-02-22T02:52:55.644 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-22T02:52:55.645 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-02-22T02:52:55.646 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-22T02:52:55.647 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:52:55.647 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-02-22T02:52:55.647 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-02-22T02:52:55.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-22T02:52:55.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-22T02:52:55.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-22T02:52:55.648 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-22T02:52:55.670 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:55.675+0000 7f37ed49a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:55.692 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:55.695+0000 7f37ed49a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:55.693 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:55.699+0000 7f37ed49a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:52:56.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-22T02:52:56.080 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:52:56.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:52:56.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:52:56.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:52:56.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:56.081 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:52:56.081 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:52:56.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:56.082 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:56.242 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:56.247+0000 7f37ed49a7c0 -1 Falling back to public interface 2024-02-22T02:52:56.544 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:57.545 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:57.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:57.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:52:57.546 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:52:57.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:57.547 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:57.966 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:52:58.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:52:58.968 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:52:58.968 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:52:58.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:52:58.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:52:58.969 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:52:59.281 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:52:59.287+0000 7f37ed49a7c0 -1 osd.0 6 log_to_monitors true 2024-02-22T02:52:59.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:00.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:00.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:00.398 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:53:00.398 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:53:00.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:00.399 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:00.840 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/295432696,v1:127.0.0.1:6803/295432696] [v2:127.0.0.1:6804/295432696,v1:127.0.0.1:6805/295432696] exists,up 0d70027c-0752-4ea2-9563-33bff9d3c69c 2024-02-22T02:53:00.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:53:00.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:53:00.840 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:53:00.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-02-22T02:53:00.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:53:00.841 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:53:00.841 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:53:00.841 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:53:00.842 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:53:00.842 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-osd.0.asok 2024-02-22T02:53:00.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-02-22T02:53:00.842 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.69653/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-02-22T02:53:00.953 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-02-22T02:53:00.954 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-02-22T02:53:00.955 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-22T02:53:00.955 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-22T02:53:00.955 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:53:00.955 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:53:00.955 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:53:00.956 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-osd.0.asok 2024-02-22T02:53:00.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-02-22T02:53:00.956 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.69653/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-02-22T02:53:01.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-02-22T02:53:01.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-02-22T02:53:01.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-02-22T02:53:01.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2024-02-22T02:53:01.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-02-22T02:53:01.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:53:01.078 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-02-22T02:53:01.079 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:53:01.079 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:53:01.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:53:01.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:53:01.079 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:53:01.187 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:53:01.187 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:53:01.188 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:53:01.189 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:53:01.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:53:01.190 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:53:01.190 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:53:01.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:53:01.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:53:01.191 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:53:01.192 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:53:01.193 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:53:01.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:53:01.194 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-02-22T02:53:01.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:53:01.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:53:01.198 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:53:01.198 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69653 2024-02-22T02:53:01.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:53:01.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:53:01.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-02-22T02:53:01.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_destroy_osd td/ceph-helpers 2024-02-22T02:53:01.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:796: test_destroy_osd: local dir=td/ceph-helpers 2024-02-22T02:53:01.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:798: test_destroy_osd: setup td/ceph-helpers 2024-02-22T02:53:01.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-02-22T02:53:01.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-02-22T02:53:01.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-02-22T02:53:01.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-22T02:53:01.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-02-22T02:53:01.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:53:01.201 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:53:01.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:53:01.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:53:01.201 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:53:01.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:53:01.202 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T02:53:01.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T02:53:01.203 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T02:53:01.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T02:53:01.204 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T02:53:01.205 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T02:53:01.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:53:01.205 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T02:53:01.206 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T02:53:01.206 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T02:53:01.207 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T02:53:01.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-22T02:53:01.208 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-02-22T02:53:01.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T02:53:01.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:53:01.209 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:53:01.209 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69653 2024-02-22T02:53:01.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T02:53:01.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T02:53:01.210 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-02-22T02:53:01.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-22T02:53:01.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:53:01.212 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:53:01.212 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69653 2024-02-22T02:53:01.214 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-22T02:53:01.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-22T02:53:01.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-22T02:53:01.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-02-22T02:53:01.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:799: test_destroy_osd: run_mon td/ceph-helpers a 2024-02-22T02:53:01.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-02-22T02:53:01.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-22T02:53:01.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-22T02:53:01.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-22T02:53:01.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-02-22T02:53:01.214 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-02-22T02:53:01.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-22T02:53:01.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:53:01.637 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:53:01.638 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:53:01.638 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:53:01.638 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:53:01.638 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:53:01.638 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-22T02:53:01.848 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-22T02:53:01.849 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-22T02:53:01.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:53:01.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:53:01.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-22T02:53:01.850 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-22T02:53:01.851 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:53:01.851 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:53:01.851 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:53:01.851 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:53:01.851 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:53:01.852 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:53:01.852 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-mon.a.asok 2024-02-22T02:53:01.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:53:01.852 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69653/ceph-mon.a.asok config get fsid 2024-02-22T02:53:01.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-22T02:53:01.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-22T02:53:01.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-22T02:53:01.968 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-22T02:53:01.969 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-22T02:53:01.969 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-22T02:53:01.969 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-22T02:53:01.969 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-22T02:53:01.970 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-22T02:53:01.970 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:53:01.970 INFO:tasks.workunit.client.0.smithi046.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:53:01.970 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69653/ceph-mon.a.asok 2024-02-22T02:53:01.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-22T02:53:01.970 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69653/ceph-mon.a.asok config get mon_host 2024-02-22T02:53:02.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:800: test_destroy_osd: run_mgr td/ceph-helpers x 2024-02-22T02:53:02.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-02-22T02:53:02.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-22T02:53:02.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-22T02:53:02.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-22T02:53:02.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-02-22T02:53:02.084 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-22T02:53:03.017 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-22T02:53:03.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:53:03.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:53:03.018 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:53:03.018 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:53:03.018 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:53:03.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:53:03.018 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:53:03.020 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-22T02:53:03.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:801: test_destroy_osd: run_osd td/ceph-helpers 0 2024-02-22T02:53:03.091 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-02-22T02:53:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-22T02:53:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-22T02:53:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-22T02:53:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-02-22T02:53:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-02-22T02:53:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-22T02:53:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-22T02:53:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-22T02:53:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-02-22T02:53:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-02-22T02:53:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-22T02:53:03.092 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-22T02:53:03.093 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-02-22T02:53:03.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-22T02:53:03.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-22T02:53:03.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-22T02:53:03.093 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-22T02:53:03.093 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T02:53:03.093 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T02:53:03.093 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:53:03.094 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' 2024-02-22T02:53:03.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-22T02:53:03.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-22T02:53:03.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-22T02:53:03.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-02-22T02:53:03.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-02-22T02:53:03.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-22T02:53:03.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-22T02:53:03.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-22T02:53:03.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-22T02:53:03.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-22T02:53:03.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-22T02:53:03.095 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-02-22T02:53:03.095 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-22T02:53:03.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c2e59abb-ed6d-46ec-b33a-d9217a2f2d0b 2024-02-22T02:53:03.096 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c2e59abb-ed6d-46ec-b33a-d9217a2f2d0b' 2024-02-22T02:53:03.096 INFO:tasks.workunit.client.0.smithi046.stdout:add osd0 c2e59abb-ed6d-46ec-b33a-d9217a2f2d0b 2024-02-22T02:53:03.096 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-22T02:53:03.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPt9Zl31Q4BxAA8P4uCgX6z4EMoAnrohHEMw== 2024-02-22T02:53:03.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPt9Zl31Q4BxAA8P4uCgX6z4EMoAnrohHEMw=="}' 2024-02-22T02:53:03.114 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c2e59abb-ed6d-46ec-b33a-d9217a2f2d0b -i td/ceph-helpers/0/new.json 2024-02-22T02:53:03.437 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:53:03.455 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-02-22T02:53:03.457 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPt9Zl31Q4BxAA8P4uCgX6z4EMoAnrohHEMw== --osd-uuid c2e59abb-ed6d-46ec-b33a-d9217a2f2d0b 2024-02-22T02:53:03.489 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:53:03.491+0000 7f721ecb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:53:03.514 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:53:03.519+0000 7f721ecb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:53:03.515 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:53:03.519+0000 7f721ecb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:53:03.516 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:53:03.519+0000 7f721ecb27c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-02-22T02:53:03.516 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:53:03.519+0000 7f721ecb27c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-02-22T02:53:03.516 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:53:03.519+0000 7f721ecb27c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-02-22T02:53:03.554 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:53:03.559+0000 7f721ecb27c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-02-22T02:53:09.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-02-22T02:53:09.911 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-22T02:53:09.912 INFO:tasks.workunit.client.0.smithi046.stdout:adding osd0 key to auth repository 2024-02-22T02:53:09.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-22T02:53:09.913 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-22T02:53:10.545 INFO:tasks.workunit.client.0.smithi046.stdout:start osd.0 2024-02-22T02:53:10.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-22T02:53:10.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-22T02:53:10.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-22T02:53:10.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-22T02:53:10.546 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bbdca875-8fa4-4ed8-9bc6-50976bc41981 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69653/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-22T02:53:10.579 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:53:10.582+0000 7f88888427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:53:10.595 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:53:10.598+0000 7f88888427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:53:10.596 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:53:10.598+0000 7f88888427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-22T02:53:10.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-22T02:53:10.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-22T02:53:10.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-22T02:53:10.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-22T02:53:10.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-22T02:53:10.977 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:10.978 INFO:tasks.workunit.client.0.smithi046.stdout:0 2024-02-22T02:53:10.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-22T02:53:10.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:10.978 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:11.164 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:53:11.166+0000 7f88888427c0 -1 Falling back to public interface 2024-02-22T02:53:11.395 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:12.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:12.396 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:12.397 INFO:tasks.workunit.client.0.smithi046.stdout:1 2024-02-22T02:53:12.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-22T02:53:12.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:12.397 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:12.820 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:13.735 INFO:tasks.workunit.client.0.smithi046.stderr:2024-02-22T02:53:13.738+0000 7f88888427c0 -1 osd.0 0 log_to_monitors true 2024-02-22T02:53:13.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:13.821 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:13.821 INFO:tasks.workunit.client.0.smithi046.stdout:2 2024-02-22T02:53:13.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-22T02:53:13.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:13.822 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:14.340 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:15.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:15.342 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:15.342 INFO:tasks.workunit.client.0.smithi046.stdout:3 2024-02-22T02:53:15.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-22T02:53:15.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:15.343 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:15.780 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:16.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:16.782 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:16.782 INFO:tasks.workunit.client.0.smithi046.stdout:4 2024-02-22T02:53:16.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-22T02:53:16.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:16.783 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:17.218 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:18.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:18.220 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:18.220 INFO:tasks.workunit.client.0.smithi046.stdout:5 2024-02-22T02:53:18.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-22T02:53:18.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:18.221 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:18.735 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:19.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:19.737 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:19.738 INFO:tasks.workunit.client.0.smithi046.stdout:6 2024-02-22T02:53:19.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-22T02:53:19.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:19.738 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:20.200 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:21.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:21.202 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:21.203 INFO:tasks.workunit.client.0.smithi046.stdout:7 2024-02-22T02:53:21.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-22T02:53:21.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:21.203 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:21.867 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:22.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:22.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:22.869 INFO:tasks.workunit.client.0.smithi046.stdout:8 2024-02-22T02:53:22.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-22T02:53:22.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:22.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:23.301 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:24.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:24.303 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:24.303 INFO:tasks.workunit.client.0.smithi046.stdout:9 2024-02-22T02:53:24.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-22T02:53:24.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:24.304 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:24.814 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:25.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:25.816 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:25.816 INFO:tasks.workunit.client.0.smithi046.stdout:10 2024-02-22T02:53:25.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-22T02:53:25.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:25.817 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:26.236 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:27.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:27.237 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:27.238 INFO:tasks.workunit.client.0.smithi046.stdout:11 2024-02-22T02:53:27.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-22T02:53:27.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:27.238 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:28.146 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:29.148 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:29.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:29.149 INFO:tasks.workunit.client.0.smithi046.stdout:12 2024-02-22T02:53:29.149 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-22T02:53:29.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:29.150 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:29.946 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:30.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:30.947 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:30.947 INFO:tasks.workunit.client.0.smithi046.stdout:13 2024-02-22T02:53:30.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-22T02:53:30.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:30.948 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:32.060 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:33.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:33.062 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:33.062 INFO:tasks.workunit.client.0.smithi046.stdout:14 2024-02-22T02:53:33.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-22T02:53:33.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:33.063 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:33.869 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:34.870 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:34.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:34.871 INFO:tasks.workunit.client.0.smithi046.stdout:15 2024-02-22T02:53:34.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-22T02:53:34.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:34.871 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:35.665 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:36.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:36.667 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:36.667 INFO:tasks.workunit.client.0.smithi046.stdout:16 2024-02-22T02:53:36.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-22T02:53:36.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:36.668 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:37.563 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-22T02:53:38.564 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-22T02:53:38.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-22T02:53:38.565 INFO:tasks.workunit.client.0.smithi046.stdout:17 2024-02-22T02:53:38.565 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-22T02:53:38.567 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-22T02:53:38.568 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-22T02:53:39.037 INFO:tasks.workunit.client.0.smithi046.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2859380578,v1:127.0.0.1:6803/2859380578] [v2:127.0.0.1:6804/2859380578,v1:127.0.0.1:6805/2859380578] exists,up c2e59abb-ed6d-46ec-b33a-d9217a2f2d0b 2024-02-22T02:53:39.037 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-22T02:53:39.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-22T02:53:39.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-22T02:53:39.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:802: test_destroy_osd: destroy_osd td/ceph-helpers 0 2024-02-22T02:53:39.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/ceph-helpers 2024-02-22T02:53:39.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=0 2024-02-22T02:53:39.038 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.0 2024-02-22T02:53:39.611 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 is already out. 2024-02-22T02:53:39.627 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-02-22T02:53:39.627 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T02:53:39.628 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T02:53:39.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T02:53:39.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T02:53:39.628 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T02:53:39.935 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T02:53:39.936 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.0 2024-02-22T02:53:40.390 INFO:tasks.workunit.client.0.smithi046.stderr:osd.0 is already down. 2024-02-22T02:53:40.407 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2024-02-22T05:51:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-22T05:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr:Terminated 2024-02-22T05:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1: destroy_osd: teardown td/ceph-helpers 1 2024-02-22T05:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-02-22T05:51:09.887 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-02-22T05:51:09.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-02-22T05:51:09.888 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T05:51:09.888 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T05:51:09.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T05:51:09.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T05:51:09.888 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T05:51:09.976 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T05:51:09.976 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T05:51:09.977 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T05:51:09.978 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T05:51:09.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T05:51:09.979 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T05:51:09.979 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T05:51:09.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T05:51:09.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T05:51:09.981 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T05:51:09.981 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T05:51:09.982 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T05:51:09.983 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-02-22T05:51:09.983 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-02-22T05:51:09.983 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-02-22T05:51:09.984 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/ceph-helpers/mgr.x.log td/ceph-helpers/mon.a.log td/ceph-helpers/osd.0.log /home/ubuntu/cephtest/archive/log 2024-02-22T05:51:09.985 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-02-22T05:51:09.990 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T05:51:09.990 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T05:51:09.990 INFO:tasks.workunit.client.0.smithi046.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T05:51:09.990 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69653 2024-02-22T05:51:09.992 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T05:51:09.992 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T05:51:09.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: return 1 2024-02-22T05:51:09.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:802: test_destroy_osd: return 1 2024-02-22T05:51:09.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2341: run_tests: teardown td/ceph-helpers 1 2024-02-22T05:51:09.992 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-02-22T05:51:09.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-02-22T05:51:09.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-02-22T05:51:09.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-22T05:51:09.993 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-22T05:51:09.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-22T05:51:09.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-22T05:51:09.993 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-22T05:51:09.994 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-22T05:51:09.994 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-22T05:51:09.995 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-22T05:51:09.995 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-22T05:51:09.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-22T05:51:09.996 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-22T05:51:09.996 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-22T05:51:09.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T05:51:09.997 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-22T05:51:09.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-22T05:51:09.998 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-22T05:51:09.998 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-22T05:51:10.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-02-22T05:51:10.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-02-22T05:51:10.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-02-22T05:51:10.000 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv 'td/ceph-helpers/*.log' /home/ubuntu/cephtest/archive/log 2024-02-22T05:51:10.002 INFO:tasks.workunit.client.0.smithi046.stderr:mv: cannot stat 'td/ceph-helpers/*.log': No such file or directory 2024-02-22T05:51:10.002 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-02-22T05:51:10.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-22T05:51:10.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-22T05:51:10.003 INFO:tasks.workunit.client.0.smithi046.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69653 2024-02-22T05:51:10.003 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69653 2024-02-22T05:51:10.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-22T05:51:10.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-22T05:51:10.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2342: run_tests: return 1 2024-02-22T05:51:10.004 INFO:tasks.workunit.client.0.smithi046.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: : exit 1 2024-02-22T05:51:10.005 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2024-02-22T05:51:10.005 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-22T05:51:10.331 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_dbe7fcec16eddee6239e4dd68c4d0203c7df6461/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_dbe7fcec16eddee6239e4dd68c4d0203c7df6461/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_dbe7fcec16eddee6239e4dd68c4d0203c7df6461/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on smithi046 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbe7fcec16eddee6239e4dd68c4d0203c7df6461 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2024-02-22T05:51:10.568 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=231421c9494c4cbb97f24cd6d57587a5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_dbe7fcec16eddee6239e4dd68c4d0203c7df6461/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_dbe7fcec16eddee6239e4dd68c4d0203c7df6461/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_dbe7fcec16eddee6239e4dd68c4d0203c7df6461/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on smithi046 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbe7fcec16eddee6239e4dd68c4d0203c7df6461 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2024-02-22T05:51:10.571 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-22T05:51:10.580 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-22T05:51:10.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-22T05:51:10.652 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-22T05:51:10.661 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-22T05:51:10.661 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-22T05:51:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: remote refid st t when poll reach delay offset jitter 2024-02-22T05:51:10.679 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================== 2024-02-22T05:51:10.679 INFO:teuthology.orchestra.run.smithi046.stdout:*hv01.front.sepi 44.4.53.2 2 u 183 512 377 0.202 -2.402 0.405 2024-02-22T05:51:10.679 INFO:teuthology.orchestra.run.smithi046.stdout:+hv02.front.sepi 63.231.80.2 3 u 241 512 377 0.185 +0.484 0.245 2024-02-22T05:51:10.679 INFO:teuthology.orchestra.run.smithi046.stdout:+hv03.front.sepi 74.6.168.72 3 u 243 512 377 0.198 -1.299 0.153 2024-02-22T05:51:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-02-22T05:51:10.680 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-22T05:51:10.688 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-22T05:51:10.689 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-22T05:51:10.697 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-22T05:51:10.705 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-22T05:51:10.714 INFO:teuthology.task.internal:Duration was 12613.619991 seconds 2024-02-22T05:51:10.714 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-22T05:51:10.722 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-22T05:51:10.722 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-22T05:51:10.764 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-22T05:51:10.764 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-02-22T05:51:10.764 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-22T05:51:10.820 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-22T05:51:10.821 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-22T05:51:10.892 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-22T05:51:10.892 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-22T05:51:10.991 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-22T05:51:11.000 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-22T05:51:11.000 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-22T05:51:11.013 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-22T05:51:11.025 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-22T05:51:11.065 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-02-22T05:51:11.082 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-22T05:51:11.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-22T05:51:11.127 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-22T05:51:11.137 INFO:teuthology.task.internal:Transferring archived files... 2024-02-22T05:51:11.138 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdhange-2024-02-22_01:13:10-rados-wip-pdhange-testing1-distro-default-smithi/7570447/remote/smithi046 2024-02-22T05:51:11.139 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-22T05:51:25.911 INFO:teuthology.task.internal:Removing archive directory... 2024-02-22T05:51:25.911 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-22T05:51:25.985 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-22T05:51:25.996 INFO:teuthology.task.internal:Not uploading archives. 2024-02-22T05:51:25.996 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-22T05:51:26.017 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-22T05:51:26.018 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-22T05:51:26.024 INFO:teuthology.orchestra.run.smithi046.stdout: 658642 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 22 05:51 /home/ubuntu/cephtest 2024-02-22T05:51:26.024 INFO:teuthology.orchestra.run.smithi046.stdout: 671383 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 22 02:30 /home/ubuntu/cephtest/mnt.0 2024-02-22T05:51:26.025 INFO:teuthology.orchestra.run.smithi046.stdout: 673410 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 22 02:51 /home/ubuntu/cephtest/mnt.0/client.0 2024-02-22T05:51:26.025 INFO:teuthology.orchestra.run.smithi046.stdout: 688454 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 22 02:51 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-22T05:51:26.025 INFO:teuthology.orchestra.run.smithi046.stdout: 688455 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Feb 22 05:51 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-02-22T05:51:26.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-22T05:51:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-22T05:51:26.026 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-22T05:51:26.032 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-22T05:51:26.043 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-22T05:51:26.053 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi046 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-22T05:51:26.053 INFO:teuthology.nuke:Checking targets against current locks 2024-02-22T05:51:26.092 DEBUG:teuthology.nuke:shortname: smithi046 2024-02-22T05:51:26.093 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-22T05:51:26.129 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdhange-2024-02-22_01:13:10-rados-wip-pdhange-testing1-distro-default-smithi/7570447', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-22 02:11:32.209018', 'locked_by': 'scheduled_pdhange@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPheQoPgX42Hb+CI14rS5w8sLfs7Knm3a9aQ03qEo1ZQxQqySdR4ER39y7uTAbr58u9LK7gREjfm9xOj+WnCLVA='} 2024-02-22T05:51:26.216 INFO:teuthology.orchestra.console.smithi046:Power off 2024-02-22T05:51:26.216 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-22T05:51:26.277 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2024-02-22T05:51:26.278 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-22T05:51:26.333 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-02-22T05:51:30.334 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-22T05:51:30.445 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-02-22T05:51:34.448 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-22T05:51:34.559 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-02-22T05:51:38.562 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-22T05:51:38.673 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2024-02-22T05:51:38.774 INFO:teuthology.orchestra.console.smithi046:Power off completed 2024-02-22T05:51:38.879 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} duration: 12613.619990825653 failure_reason: 'Command failed (workunit test misc/test-ceph-helpers.sh) on smithi046 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbe7fcec16eddee6239e4dd68c4d0203c7df6461 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh''' flavor: default owner: scheduled_pdhange@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=231421c9494c4cbb97f24cd6d57587a5 status: fail success: false 2024-02-22T05:51:38.880 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-22T05:51:38.991 INFO:teuthology.run:FAIL